Bulk Core/Load CDK: Support for connector-type-specific non-config sp… #45463
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
johnny-schmidt
force-pushed
the
issue-9365/destination-spec-extension
branch
from
d569b01 to
66ee962
Compare
There was a problem hiding this comment.
seems like a reasonable implementation. nonblocking question (feel free to merge as-is): do we want to keep this in-code, or do we eventually want to parse these out of some resource file?
(it's convenient that I can currently look at <path-to-connector>/spec.json for any connector, but I don't really care that much. Maybe we're better off having the code be source of truth, but autogen some spec.json equivalent for human reference?)
| } | ||
| } | ||
|
|
||
| interface SpecificationExtender : (ConnectorSpecification) -> ConnectorSpecification |
There was a problem hiding this comment.
TIL you can do this. that's cool!
There was a problem hiding this comment.
Yes @postamar suggested that (in a different context). It's cleaner than using functions as arguments and allows you to extend/inject as needed.
There was a problem hiding this comment.
I wonder if this may or should be a fun interface. I don't know. I'm not sure it matters either.
| @@ -15,13 +16,24 @@ import java.net.URI | |||
| class SpecOperation( | |||
| @Value("\${airbyte.connector.metadata.documentation-url}") val documentationUrl: String, | |||
| val configJsonObjectSupplier: ConfigurationJsonObjectSupplier<*>, | |||
| val specificationExtender: SpecificationExtender, | |||
There was a problem hiding this comment.
naming nit: given that we're calling this as a function, can we rename to a verb? extendSpecification or something
johnny-schmidt
force-pushed
the
issue-9365/destination-spec-extension
branch
from
66ee962 to
817cc46
Compare
|
I forgot to thank you for this! Thank you, this is neat. |
|
We'll need this to transform the JSON schema AST too. It's not something that I look forward to but it's necessary to deal with |
What
Adds general support for extending the generated specification to include values specific to the destination type but not available on the metadata or the configuration json.
This is specifically to meet destination needs for expressing feature-specific things like support for refreshes.
Example usage is here
How
Core: Adds an injection point for a specification extender and a default identity implementation
Load: Adds a destination-specific implementation of the extender that applies fields from a (required) injected destination specification.