From 37af2f013d6d51270a35ec98e97309d7351d6164 Mon Sep 17 00:00:00 2001 From: btkcodedev Date: Tue, 17 Sep 2024 04:11:34 +0530 Subject: [PATCH] source-google-tasks contribution from btkcodedev (#45427) Co-authored-by: Octavia Squidington III --- .../connectors/source-google-tasks/README.md | 52 ++++ .../acceptance-test-config.yml | 17 + .../connectors/source-google-tasks/icon.svg | 6 + .../source-google-tasks/manifest.yaml | 291 ++++++++++++++++++ .../source-google-tasks/metadata.yaml | 35 +++ docs/integrations/sources/google-tasks.md | 48 +++ 6 files changed, 449 insertions(+) create mode 100644 airbyte-integrations/connectors/source-google-tasks/README.md create mode 100644 airbyte-integrations/connectors/source-google-tasks/acceptance-test-config.yml create mode 100644 airbyte-integrations/connectors/source-google-tasks/icon.svg create mode 100644 airbyte-integrations/connectors/source-google-tasks/manifest.yaml create mode 100644 airbyte-integrations/connectors/source-google-tasks/metadata.yaml create mode 100644 docs/integrations/sources/google-tasks.md diff --git a/airbyte-integrations/connectors/source-google-tasks/README.md b/airbyte-integrations/connectors/source-google-tasks/README.md new file mode 100644 index 000000000000..5d007bf36a0e --- /dev/null +++ b/airbyte-integrations/connectors/source-google-tasks/README.md @@ -0,0 +1,52 @@ +# Google Tasks +This directory contains the manifest-only connector for `source-google-tasks`. + +## Documentation reference: +Visit `https://developers.google.com/tasks/reference/rest` for API documentation + +## Authentication setup +`Source-productive` uses bearer token authentication, +Visit `https://support.google.com/googleapi/answer/6158849?hl=en&ref_topic=7013279` for getting bearer token via OAuth2.0 + +## Setting postman for getting bearer token +Currently Code granted OAuth 2.0 is not directly supported by airbyte, thus you could setup postman for getting the bearer token which could be used as `api_key`, +Steps: +- Visit google cloud `https://console.cloud.google.com/apis/api/tasks.googleapis.com/metrics` and enable the tasks api service +- Go to the consent screen `https://console.cloud.google.com/apis/credentials/consent` and add your email for enabling postman testing access +- Visit `https://console.cloud.google.com/apis/credentials` and create new credentails for OAuth 2.0 and copy client id and client secret +- Add callback url `https://oauth.pstmn.io/v1/callback` while credential creation +- Goto postman client and select new tab for setting authorization to OAuth 2.0 + - Set scope as `https://www.googleapis.com/auth/tasks https://www.googleapis.com/auth/tasks.readonly` + - Set access token URL as `https://accounts.google.com/o/oauth2/token` + - Set auth URL as `https://accounts.google.com/o/oauth2/v2/auth` + - Click `Get New Access Token` and authorize via your google account + - Copy the resulted bearer token and use it as credential for the connector + +## Usage +There are multiple ways to use this connector: +- You can use this connector as any other connector in Airbyte Marketplace. +- You can load this connector in `pyairbyte` using `get_source`! +- You can open this connector in Connector Builder, edit it, and publish to your workspaces. + +Please refer to the manifest-only connector documentation for more details. + +## Local Development +We recommend you use the Connector Builder to edit this connector. + +But, if you want to develop this connector locally, you can use the following steps. + +### Environment Setup +You will need `airbyte-ci` installed. You can find the documentation [here](airbyte-ci). + +### Build +This will create a dev image (`source-google-tasks:dev`) that you can use to test the connector locally. +```bash +airbyte-ci connectors --name=source-google-tasks build +``` + +### Test +This will run the acceptance tests for the connector. +```bash +airbyte-ci connectors --name=source-google-tasks test +``` + diff --git a/airbyte-integrations/connectors/source-google-tasks/acceptance-test-config.yml b/airbyte-integrations/connectors/source-google-tasks/acceptance-test-config.yml new file mode 100644 index 000000000000..3ef106927c86 --- /dev/null +++ b/airbyte-integrations/connectors/source-google-tasks/acceptance-test-config.yml @@ -0,0 +1,17 @@ +# See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) +# for more information about how to configure these tests +connector_image: airbyte/source-google-tasks:dev +acceptance_tests: + spec: + tests: + - spec_path: "manifest.yaml" + connection: + bypass_reason: "This is a builder contribution, and we do not have secrets at this time" + discovery: + bypass_reason: "This is a builder contribution, and we do not have secrets at this time" + basic_read: + bypass_reason: "This is a builder contribution, and we do not have secrets at this time" + incremental: + bypass_reason: "This is a builder contribution, and we do not have secrets at this time" + full_refresh: + bypass_reason: "This is a builder contribution, and we do not have secrets at this time" diff --git a/airbyte-integrations/connectors/source-google-tasks/icon.svg b/airbyte-integrations/connectors/source-google-tasks/icon.svg new file mode 100644 index 000000000000..f81301bb8321 --- /dev/null +++ b/airbyte-integrations/connectors/source-google-tasks/icon.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/airbyte-integrations/connectors/source-google-tasks/manifest.yaml b/airbyte-integrations/connectors/source-google-tasks/manifest.yaml new file mode 100644 index 000000000000..4331136059dc --- /dev/null +++ b/airbyte-integrations/connectors/source-google-tasks/manifest.yaml @@ -0,0 +1,291 @@ +version: 4.6.2 + +type: DeclarativeSource + +description: >- + Website: https://tasksboard.com/app + + API Docs: https://developers.google.com/tasks/reference/rest + + Auth Docs: + https://support.google.com/googleapi/answer/6158849?hl=en&ref_topic=7013279 + +check: + type: CheckStream + stream_names: + - tasks + +definitions: + streams: + tasks: + type: DeclarativeStream + name: tasks + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: pageToken + inject_into: request_parameter + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('nextPageToken') }}" + stop_condition: "{{ not response.get('nextPageToken') }}" + requester: + $ref: "#/definitions/base_requester" + path: users/@me/lists + http_method: GET + error_handler: + type: CompositeErrorHandler + error_handlers: + - type: DefaultErrorHandler + max_retries: 5 + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + error_message: Rate limit hit + backoff_strategies: + - type: ExponentialBackoffStrategy + factor: 2 + request_parameters: + maxResults: "{{ config['records_limit'] }}" + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - items + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/tasks" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + end_datetime: + type: MinMaxDatetime + datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}" + datetime_format: "%Y-%m-%dT%H:%M:%SZ" + start_datetime: + type: MinMaxDatetime + datetime: '{{ config["start_date"] }}' + datetime_format: "%Y-%m-%dT%H:%M:%SZ" + datetime_format: "%Y-%m-%dT%H:%M:%S.%fZ" + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%fZ" + lists_tasks: + type: DeclarativeStream + name: lists_tasks + retriever: + type: SimpleRetriever + paginator: + type: DefaultPaginator + page_token_option: + type: RequestOption + field_name: pageToken + inject_into: request_parameter + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('nextPageToken') }}" + stop_condition: "{{ not response.get('nextPageToken') }}" + requester: + $ref: "#/definitions/base_requester" + path: lists/{{ stream_partition.tasklist_id }}/tasks + http_method: GET + error_handler: + type: CompositeErrorHandler + error_handlers: + - type: DefaultErrorHandler + max_retries: 5 + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + error_message: Rate limit hit + backoff_strategies: + - type: ExponentialBackoffStrategy + factor: 2 + request_parameters: + maxResults: "{{ config['records_limit'] }}" + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - items + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + stream: + $ref: "#/definitions/streams/tasks" + parent_key: id + partition_field: tasklist_id + primary_key: + - id + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/lists_tasks" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + end_datetime: + type: MinMaxDatetime + datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}" + datetime_format: "%Y-%m-%dT%H:%M:%SZ" + start_datetime: + type: MinMaxDatetime + datetime: '{{ config["start_date"] }}' + datetime_format: "%Y-%m-%dT%H:%M:%SZ" + datetime_format: "%Y-%m-%dT%H:%M:%S.%fZ" + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%fZ" + base_requester: + type: HttpRequester + url_base: https://tasks.googleapis.com/tasks/v1/ + authenticator: + type: BearerAuthenticator + api_token: '{{ config["api_key"] }}' + +streams: + - $ref: "#/definitions/streams/tasks" + - $ref: "#/definitions/streams/lists_tasks" + +spec: + type: Spec + connection_specification: + type: object + $schema: http://json-schema.org/draft-07/schema# + required: + - api_key + - start_date + properties: + api_key: + type: string + order: 1 + title: API Key + airbyte_secret: true + start_date: + type: string + order: 2 + title: Start date + format: date-time + pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ + records_limit: + type: string + description: The maximum number of records to be returned per request + order: 0 + title: Records Limit + default: "50" + additionalProperties: true + +metadata: + assist: + docsUrl: https://developers.google.com/tasks/reference/rest + testedStreams: + tasks: + hasRecords: true + streamHash: 06d4c0949ab53aa3ea7e089781332fea60caf2f1 + hasResponse: true + primaryKeysAreUnique: true + primaryKeysArePresent: true + responsesAreSuccessful: true + lists_tasks: + hasRecords: true + streamHash: 6d6137d9a88020f99badd5a98968cb1a12e07cdd + hasResponse: true + primaryKeysAreUnique: true + primaryKeysArePresent: true + responsesAreSuccessful: true + autoImportSchema: + tasks: true + lists_tasks: true + +schemas: + tasks: + type: object + $schema: http://json-schema.org/schema# + required: + - id + - updated + properties: + id: + type: string + etag: + type: + - string + - "null" + kind: + type: + - string + - "null" + title: + type: + - string + - "null" + updated: + type: string + selfLink: + type: + - string + - "null" + additionalProperties: true + lists_tasks: + type: object + $schema: http://json-schema.org/schema# + required: + - id + - updated + properties: + id: + type: string + due: + type: + - string + - "null" + etag: + type: + - string + - "null" + kind: + type: + - string + - "null" + links: + type: + - array + - "null" + title: + type: + - string + - "null" + status: + type: + - string + - "null" + updated: + type: string + position: + type: + - string + - "null" + selfLink: + type: + - string + - "null" + completed: + type: + - string + - "null" + webViewLink: + type: + - string + - "null" + additionalProperties: true diff --git a/airbyte-integrations/connectors/source-google-tasks/metadata.yaml b/airbyte-integrations/connectors/source-google-tasks/metadata.yaml new file mode 100644 index 000000000000..e97b8828856b --- /dev/null +++ b/airbyte-integrations/connectors/source-google-tasks/metadata.yaml @@ -0,0 +1,35 @@ +metadataSpecVersion: "1.0" +data: + allowedHosts: + hosts: + - "tasks.googleapis.com" + registryOverrides: + oss: + enabled: true + cloud: + enabled: true + remoteRegistries: + pypi: + enabled: false + packageName: airbyte-source-google-tasks + connectorBuildOptions: + baseImage: docker.io/airbyte/source-declarative-manifest:4.6.2@sha256:f5fcd3d4703b7590b6166a7853c5ed1686731607cd30a159a8c24e2fe2c1ee98 + connectorSubtype: api + connectorType: source + definitionId: 751c2519-1446-416e-9736-9b98585f8125 + dockerImageTag: 0.0.1 + dockerRepository: airbyte/source-google-tasks + githubIssueLabel: source-google-tasks + icon: icon.svg + license: MIT + name: Google Tasks + releaseDate: 2024-09-12 + releaseStage: alpha + supportLevel: community + documentationUrl: https://docs.airbyte.com/integrations/sources/google-tasks + tags: + - language:manifest-only + - cdk:low-code + ab_internal: + ql: 100 + sl: 100 diff --git a/docs/integrations/sources/google-tasks.md b/docs/integrations/sources/google-tasks.md new file mode 100644 index 000000000000..cb512a79e582 --- /dev/null +++ b/docs/integrations/sources/google-tasks.md @@ -0,0 +1,48 @@ +# Google Tasks +This page contains the setup guide and reference information for the [Google Tasks](https://tasksboard.com/app) source connector. + +## Documentation reference: +Visit `https://developers.google.com/tasks/reference/rest` for API documentation + +## Authentication setup +`Source-productive` uses bearer token authentication, +Visit `https://support.google.com/googleapi/answer/6158849?hl=en&ref_topic=7013279` for getting bearer token via OAuth2.0 + +## Setting postman for getting bearer token +Currently Code granted OAuth 2.0 is not directly supported by airbyte, thus you could setup postman for getting the bearer token which could be used as `api_key`, +Steps: +- Visit google cloud `https://console.cloud.google.com/apis/api/tasks.googleapis.com/metrics` and enable the tasks api service +- Go to the consent screen `https://console.cloud.google.com/apis/credentials/consent` and add your email for enabling postman testing access +- Visit `https://console.cloud.google.com/apis/credentials` and create new credentails for OAuth 2.0 and copy client id and client secret +- Add callback url `https://oauth.pstmn.io/v1/callback` while credential creation +- Goto postman client and select new tab for setting authorization to OAuth 2.0 + - Set scope as `https://www.googleapis.com/auth/tasks https://www.googleapis.com/auth/tasks.readonly` + - Set access token URL as `https://accounts.google.com/o/oauth2/token` + - Set auth URL as `https://accounts.google.com/o/oauth2/v2/auth` + - Click `Get New Access Token` and authorize via your google account + - Copy the resulted bearer token and use it as credential for the connector + +## Configuration + +| Input | Type | Description | Default Value | +|-------|------|-------------|---------------| +| `api_key` | `string` | API Key. | | +| `start_date` | `string` | Start date. | | +| `records_limit` | `string` | Records Limit. The maximum number of records to be returned per request | 50 | + +## Streams +| Stream Name | Primary Key | Pagination | Supports Full Sync | Supports Incremental | +|-------------|-------------|------------|---------------------|----------------------| +| tasks | id | DefaultPaginator | ✅ | ✅ | +| lists_tasks | id | DefaultPaginator | ✅ | ✅ | + +## Changelog + +
+ Expand to review + +| Version | Date |PR| Subject | +|------------------|------------|---|----------------| +| 0.0.1 | 2024-09-12 |[45427](https://github.com/airbytehq/airbyte/pull/45427)| Initial release by [@btkcodedev](https://github.com/btkcodedev) via Connector Builder| + +
\ No newline at end of file