Support Per Sink Configuration of Input Feeds

[ricky-galvao]

It would be useful to permit the exclusion of a sink based on an incoming request's scope which could be defined by vendor, namespace, and version used in schema configuration. The current implementation will attempt to forward payload from requests to all configured sinks with deliveryRequired enabled. I propose a modification that would permit a list of exclusions for the sink.

Configuration in the config.yml for a sink would become something along the lines of:

sinks:
- name: sink1
   type: kinesis-firehose
   deliveryRequired: true
   defaultOuput: firehose_dest
   deadletterOutput: firehose_deadletter
   exclude:
     - vendor: vendor1
        namespace: 
          - namespace1
          - namespace2
        version:
          - v1.0
          - v1.1

An inclusion approach might also be appropriate. If include is not defined then make the sink applicable to all inputs (so as not to break current implementations), but if defined then only apply to specified filter.

[jakthom]

Ohh I very much agree! Sink-specific payload filtering would be awesome. Let's dig into this...

[jakthom]

@ricky-galvao what do you think about using https://github.com/timbray/quamina for pattern matching? It's built to handle massive volume, from Tim Bray (the guy who implemented EventBridge pattern matching at AWS).