-
Notifications
You must be signed in to change notification settings - Fork 276
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore: clean up the config:init folder. (#13165)
Co-authored-by: Pedro S. Lopez <[email protected]>
- Loading branch information
1 parent
c39711c
commit 763603f
Showing
10 changed files
with
6 additions
and
50 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,5 @@ | ||
# airbyte-config:init | ||
|
||
This module fulfills two responsibilities: | ||
1. It is where we declare what connectors should ship with the Platform. See below for more instruction on how it works. | ||
1. It is where we declare icons for loading connector definitions. This will soon be deprecated and moved entirely to the metadata service. | ||
2. It contains the scripts and Dockerfile that allow the `docker-compose` version of Airbyte to mount the local filesystem. This is helpful in cases where a user wants to use a connector that interacts with (reads data from or writes data to) the local filesystem. e.g. `destination-local-json`. | ||
|
||
## Declaring connectors that ship with the Platform | ||
In order to have a connector ship with the Platform it must be present in the respective `source_definitions.yaml` or `destination_definitions.yaml` files in `src/main/resources/seed`. If a connector is added there, the build system will handle fetching its spec and adding it to `source_specs.yaml` or `destination_specs.yaml`. See the gradle tasks to understand how this all works. The logic for fetching the specs is in `airbyte-config:specs`. |
Empty file.
Empty file.
Empty file.
8 changes: 0 additions & 8 deletions
8
...ain/resources/config/STANDARD_SOURCE_DEFINITION/2817b3f0-04e4-4c7a-9f32-7a5e8a83db95.json
This file was deleted.
Oops, something went wrong.
8 changes: 0 additions & 8 deletions
8
...ain/resources/config/STANDARD_SOURCE_DEFINITION/6fe89830-d04d-401b-aad6-6552ffa5c4af.json
This file was deleted.
Oops, something went wrong.
8 changes: 0 additions & 8 deletions
8
...ain/resources/config/STANDARD_SOURCE_DEFINITION/7e20ce3e-d820-4327-ad7a-88f3927fd97a.json
This file was deleted.
Oops, something went wrong.
8 changes: 0 additions & 8 deletions
8
...ain/resources/config/STANDARD_SOURCE_DEFINITION/c47d6804-8b98-449f-970a-5ddb5cb5d7aa.json
This file was deleted.
Oops, something went wrong.
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,16 +1,7 @@ | ||
# Generating Seed Connector Specs | ||
# Downloading Local Connector Registry | ||
|
||
The catalog of seeded connector definitions is stored and manually updated in the `airbyte-config/init/src/main/resources/seed/*_definitions.yaml` | ||
files. These manually-maintained connector definitions intentionally _do not_ contain the connector specs, in an effort to keep these files | ||
human-readable and easily-editable, and because specs can be automatically fetched. | ||
The local connector registry is downloaded from the metadata service using the `downloadConnectorRegistry` task defined in this module. | ||
|
||
This automatic fetching of connector specs is the goal of the SeedConnectorSpecGenerator. This class reads the connector definitions in | ||
the `airbyte-config/init/src/main/resources/seed/*_definitions.yaml` files, fetches the corresponding specs from the GCS bucket cache, and writes the | ||
specs to the `airbyte-config/init/src/main/resources/seed/*_specs.yaml` files. See the | ||
[SeedConnectorSpecGenerator](src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java) class for more details. | ||
|
||
Therefore, whenever a connector definition is updated in the `airbyte-config/init/src/main/resources/seed/*_definitions.yaml` files, the | ||
SeedConnectorSpecGenerator should be re-ran to generate the updated connector specs files. To do so, | ||
run `./gradlew :oss:airbyte-config:init:processResources`, or just build the platform project, and commit the changes to your PR. If you do not do this, | ||
the build in the CI will fail because there will be a diff in the generated files as you have not checked in the changes that were applied by the | ||
generator. | ||
This task is automatically run when Airbyte is built. The downloaded specs are shipped with Airbyte to account for air-gapped airbyte | ||
deployments. This also allows Airbyte to start with pre-populated connectors. After initialization, we expect most deployments to query the remote | ||
metadata service for the most up-to-date connector versions. |