Skip to content

Commit

Permalink
chore: clean up the config:init folder. (#13165)
Browse files Browse the repository at this point in the history
Co-authored-by: Pedro S. Lopez <[email protected]>
  • Loading branch information
davinchia and pedroslopez committed Jul 18, 2024
1 parent c39711c commit 763603f
Show file tree
Hide file tree
Showing 10 changed files with 6 additions and 50 deletions.
5 changes: 1 addition & 4 deletions airbyte-config/init/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
# airbyte-config:init

This module fulfills two responsibilities:
1. It is where we declare what connectors should ship with the Platform. See below for more instruction on how it works.
1. It is where we declare icons for loading connector definitions. This will soon be deprecated and moved entirely to the metadata service.
2. It contains the scripts and Dockerfile that allow the `docker-compose` version of Airbyte to mount the local filesystem. This is helpful in cases where a user wants to use a connector that interacts with (reads data from or writes data to) the local filesystem. e.g. `destination-local-json`.

## Declaring connectors that ship with the Platform
In order to have a connector ship with the Platform it must be present in the respective `source_definitions.yaml` or `destination_definitions.yaml` files in `src/main/resources/seed`. If a connector is added there, the build system will handle fetching its spec and adding it to `source_specs.yaml` or `destination_specs.yaml`. See the gradle tasks to understand how this all works. The logic for fetching the specs is in `airbyte-config:specs`.
Empty file.
Empty file.
Empty file.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

Empty file.
19 changes: 5 additions & 14 deletions airbyte-config/specs/README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,7 @@
# Generating Seed Connector Specs
# Downloading Local Connector Registry

The catalog of seeded connector definitions is stored and manually updated in the `airbyte-config/init/src/main/resources/seed/*_definitions.yaml`
files. These manually-maintained connector definitions intentionally _do not_ contain the connector specs, in an effort to keep these files
human-readable and easily-editable, and because specs can be automatically fetched.
The local connector registry is downloaded from the metadata service using the `downloadConnectorRegistry` task defined in this module.

This automatic fetching of connector specs is the goal of the SeedConnectorSpecGenerator. This class reads the connector definitions in
the `airbyte-config/init/src/main/resources/seed/*_definitions.yaml` files, fetches the corresponding specs from the GCS bucket cache, and writes the
specs to the `airbyte-config/init/src/main/resources/seed/*_specs.yaml` files. See the
[SeedConnectorSpecGenerator](src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java) class for more details.

Therefore, whenever a connector definition is updated in the `airbyte-config/init/src/main/resources/seed/*_definitions.yaml` files, the
SeedConnectorSpecGenerator should be re-ran to generate the updated connector specs files. To do so,
run `./gradlew :oss:airbyte-config:init:processResources`, or just build the platform project, and commit the changes to your PR. If you do not do this,
the build in the CI will fail because there will be a diff in the generated files as you have not checked in the changes that were applied by the
generator.
This task is automatically run when Airbyte is built. The downloaded specs are shipped with Airbyte to account for air-gapped airbyte
deployments. This also allows Airbyte to start with pre-populated connectors. After initialization, we expect most deployments to query the remote
metadata service for the most up-to-date connector versions.

0 comments on commit 763603f

Please sign in to comment.