-
Notifications
You must be signed in to change notification settings - Fork 4.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
source-google-tasks contribution from btkcodedev (#45427)
Co-authored-by: Octavia Squidington III <[email protected]>
- Loading branch information
1 parent
a85de6f
commit 37af2f0
Showing
6 changed files
with
449 additions
and
0 deletions.
There are no files selected for viewing
52 changes: 52 additions & 0 deletions
52
airbyte-integrations/connectors/source-google-tasks/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
# Google Tasks | ||
This directory contains the manifest-only connector for `source-google-tasks`. | ||
|
||
## Documentation reference: | ||
Visit `https://developers.google.com/tasks/reference/rest` for API documentation | ||
|
||
## Authentication setup | ||
`Source-productive` uses bearer token authentication, | ||
Visit `https://support.google.com/googleapi/answer/6158849?hl=en&ref_topic=7013279` for getting bearer token via OAuth2.0 | ||
|
||
## Setting postman for getting bearer token | ||
Currently Code granted OAuth 2.0 is not directly supported by airbyte, thus you could setup postman for getting the bearer token which could be used as `api_key`, | ||
Steps: | ||
- Visit google cloud `https://console.cloud.google.com/apis/api/tasks.googleapis.com/metrics` and enable the tasks api service | ||
- Go to the consent screen `https://console.cloud.google.com/apis/credentials/consent` and add your email for enabling postman testing access | ||
- Visit `https://console.cloud.google.com/apis/credentials` and create new credentails for OAuth 2.0 and copy client id and client secret | ||
- Add callback url `https://oauth.pstmn.io/v1/callback` while credential creation | ||
- Goto postman client and select new tab for setting authorization to OAuth 2.0 | ||
- Set scope as `https://www.googleapis.com/auth/tasks https://www.googleapis.com/auth/tasks.readonly` | ||
- Set access token URL as `https://accounts.google.com/o/oauth2/token` | ||
- Set auth URL as `https://accounts.google.com/o/oauth2/v2/auth` | ||
- Click `Get New Access Token` and authorize via your google account | ||
- Copy the resulted bearer token and use it as credential for the connector | ||
|
||
## Usage | ||
There are multiple ways to use this connector: | ||
- You can use this connector as any other connector in Airbyte Marketplace. | ||
- You can load this connector in `pyairbyte` using `get_source`! | ||
- You can open this connector in Connector Builder, edit it, and publish to your workspaces. | ||
|
||
Please refer to the manifest-only connector documentation for more details. | ||
|
||
## Local Development | ||
We recommend you use the Connector Builder to edit this connector. | ||
|
||
But, if you want to develop this connector locally, you can use the following steps. | ||
|
||
### Environment Setup | ||
You will need `airbyte-ci` installed. You can find the documentation [here](airbyte-ci). | ||
|
||
### Build | ||
This will create a dev image (`source-google-tasks:dev`) that you can use to test the connector locally. | ||
```bash | ||
airbyte-ci connectors --name=source-google-tasks build | ||
``` | ||
|
||
### Test | ||
This will run the acceptance tests for the connector. | ||
```bash | ||
airbyte-ci connectors --name=source-google-tasks test | ||
``` | ||
|
17 changes: 17 additions & 0 deletions
17
airbyte-integrations/connectors/source-google-tasks/acceptance-test-config.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
# See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) | ||
# for more information about how to configure these tests | ||
connector_image: airbyte/source-google-tasks:dev | ||
acceptance_tests: | ||
spec: | ||
tests: | ||
- spec_path: "manifest.yaml" | ||
connection: | ||
bypass_reason: "This is a builder contribution, and we do not have secrets at this time" | ||
discovery: | ||
bypass_reason: "This is a builder contribution, and we do not have secrets at this time" | ||
basic_read: | ||
bypass_reason: "This is a builder contribution, and we do not have secrets at this time" | ||
incremental: | ||
bypass_reason: "This is a builder contribution, and we do not have secrets at this time" | ||
full_refresh: | ||
bypass_reason: "This is a builder contribution, and we do not have secrets at this time" |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
291 changes: 291 additions & 0 deletions
291
airbyte-integrations/connectors/source-google-tasks/manifest.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,291 @@ | ||
version: 4.6.2 | ||
|
||
type: DeclarativeSource | ||
|
||
description: >- | ||
Website: https://tasksboard.com/app | ||
API Docs: https://developers.google.com/tasks/reference/rest | ||
Auth Docs: | ||
https://support.google.com/googleapi/answer/6158849?hl=en&ref_topic=7013279 | ||
check: | ||
type: CheckStream | ||
stream_names: | ||
- tasks | ||
|
||
definitions: | ||
streams: | ||
tasks: | ||
type: DeclarativeStream | ||
name: tasks | ||
retriever: | ||
type: SimpleRetriever | ||
paginator: | ||
type: DefaultPaginator | ||
page_token_option: | ||
type: RequestOption | ||
field_name: pageToken | ||
inject_into: request_parameter | ||
pagination_strategy: | ||
type: CursorPagination | ||
cursor_value: "{{ response.get('nextPageToken') }}" | ||
stop_condition: "{{ not response.get('nextPageToken') }}" | ||
requester: | ||
$ref: "#/definitions/base_requester" | ||
path: users/@me/lists | ||
http_method: GET | ||
error_handler: | ||
type: CompositeErrorHandler | ||
error_handlers: | ||
- type: DefaultErrorHandler | ||
max_retries: 5 | ||
response_filters: | ||
- type: HttpResponseFilter | ||
action: RATE_LIMITED | ||
http_codes: | ||
- 429 | ||
error_message: Rate limit hit | ||
backoff_strategies: | ||
- type: ExponentialBackoffStrategy | ||
factor: 2 | ||
request_parameters: | ||
maxResults: "{{ config['records_limit'] }}" | ||
record_selector: | ||
type: RecordSelector | ||
extractor: | ||
type: DpathExtractor | ||
field_path: | ||
- items | ||
primary_key: | ||
- id | ||
schema_loader: | ||
type: InlineSchemaLoader | ||
schema: | ||
$ref: "#/schemas/tasks" | ||
incremental_sync: | ||
type: DatetimeBasedCursor | ||
cursor_field: updated | ||
end_datetime: | ||
type: MinMaxDatetime | ||
datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}" | ||
datetime_format: "%Y-%m-%dT%H:%M:%SZ" | ||
start_datetime: | ||
type: MinMaxDatetime | ||
datetime: '{{ config["start_date"] }}' | ||
datetime_format: "%Y-%m-%dT%H:%M:%SZ" | ||
datetime_format: "%Y-%m-%dT%H:%M:%S.%fZ" | ||
cursor_datetime_formats: | ||
- "%Y-%m-%dT%H:%M:%S.%fZ" | ||
lists_tasks: | ||
type: DeclarativeStream | ||
name: lists_tasks | ||
retriever: | ||
type: SimpleRetriever | ||
paginator: | ||
type: DefaultPaginator | ||
page_token_option: | ||
type: RequestOption | ||
field_name: pageToken | ||
inject_into: request_parameter | ||
pagination_strategy: | ||
type: CursorPagination | ||
cursor_value: "{{ response.get('nextPageToken') }}" | ||
stop_condition: "{{ not response.get('nextPageToken') }}" | ||
requester: | ||
$ref: "#/definitions/base_requester" | ||
path: lists/{{ stream_partition.tasklist_id }}/tasks | ||
http_method: GET | ||
error_handler: | ||
type: CompositeErrorHandler | ||
error_handlers: | ||
- type: DefaultErrorHandler | ||
max_retries: 5 | ||
response_filters: | ||
- type: HttpResponseFilter | ||
action: RATE_LIMITED | ||
http_codes: | ||
- 429 | ||
error_message: Rate limit hit | ||
backoff_strategies: | ||
- type: ExponentialBackoffStrategy | ||
factor: 2 | ||
request_parameters: | ||
maxResults: "{{ config['records_limit'] }}" | ||
record_selector: | ||
type: RecordSelector | ||
extractor: | ||
type: DpathExtractor | ||
field_path: | ||
- items | ||
partition_router: | ||
type: SubstreamPartitionRouter | ||
parent_stream_configs: | ||
- type: ParentStreamConfig | ||
stream: | ||
$ref: "#/definitions/streams/tasks" | ||
parent_key: id | ||
partition_field: tasklist_id | ||
primary_key: | ||
- id | ||
schema_loader: | ||
type: InlineSchemaLoader | ||
schema: | ||
$ref: "#/schemas/lists_tasks" | ||
incremental_sync: | ||
type: DatetimeBasedCursor | ||
cursor_field: updated | ||
end_datetime: | ||
type: MinMaxDatetime | ||
datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}" | ||
datetime_format: "%Y-%m-%dT%H:%M:%SZ" | ||
start_datetime: | ||
type: MinMaxDatetime | ||
datetime: '{{ config["start_date"] }}' | ||
datetime_format: "%Y-%m-%dT%H:%M:%SZ" | ||
datetime_format: "%Y-%m-%dT%H:%M:%S.%fZ" | ||
cursor_datetime_formats: | ||
- "%Y-%m-%dT%H:%M:%S.%fZ" | ||
base_requester: | ||
type: HttpRequester | ||
url_base: https://tasks.googleapis.com/tasks/v1/ | ||
authenticator: | ||
type: BearerAuthenticator | ||
api_token: '{{ config["api_key"] }}' | ||
|
||
streams: | ||
- $ref: "#/definitions/streams/tasks" | ||
- $ref: "#/definitions/streams/lists_tasks" | ||
|
||
spec: | ||
type: Spec | ||
connection_specification: | ||
type: object | ||
$schema: http://json-schema.org/draft-07/schema# | ||
required: | ||
- api_key | ||
- start_date | ||
properties: | ||
api_key: | ||
type: string | ||
order: 1 | ||
title: API Key | ||
airbyte_secret: true | ||
start_date: | ||
type: string | ||
order: 2 | ||
title: Start date | ||
format: date-time | ||
pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ | ||
records_limit: | ||
type: string | ||
description: The maximum number of records to be returned per request | ||
order: 0 | ||
title: Records Limit | ||
default: "50" | ||
additionalProperties: true | ||
|
||
metadata: | ||
assist: | ||
docsUrl: https://developers.google.com/tasks/reference/rest | ||
testedStreams: | ||
tasks: | ||
hasRecords: true | ||
streamHash: 06d4c0949ab53aa3ea7e089781332fea60caf2f1 | ||
hasResponse: true | ||
primaryKeysAreUnique: true | ||
primaryKeysArePresent: true | ||
responsesAreSuccessful: true | ||
lists_tasks: | ||
hasRecords: true | ||
streamHash: 6d6137d9a88020f99badd5a98968cb1a12e07cdd | ||
hasResponse: true | ||
primaryKeysAreUnique: true | ||
primaryKeysArePresent: true | ||
responsesAreSuccessful: true | ||
autoImportSchema: | ||
tasks: true | ||
lists_tasks: true | ||
|
||
schemas: | ||
tasks: | ||
type: object | ||
$schema: http://json-schema.org/schema# | ||
required: | ||
- id | ||
- updated | ||
properties: | ||
id: | ||
type: string | ||
etag: | ||
type: | ||
- string | ||
- "null" | ||
kind: | ||
type: | ||
- string | ||
- "null" | ||
title: | ||
type: | ||
- string | ||
- "null" | ||
updated: | ||
type: string | ||
selfLink: | ||
type: | ||
- string | ||
- "null" | ||
additionalProperties: true | ||
lists_tasks: | ||
type: object | ||
$schema: http://json-schema.org/schema# | ||
required: | ||
- id | ||
- updated | ||
properties: | ||
id: | ||
type: string | ||
due: | ||
type: | ||
- string | ||
- "null" | ||
etag: | ||
type: | ||
- string | ||
- "null" | ||
kind: | ||
type: | ||
- string | ||
- "null" | ||
links: | ||
type: | ||
- array | ||
- "null" | ||
title: | ||
type: | ||
- string | ||
- "null" | ||
status: | ||
type: | ||
- string | ||
- "null" | ||
updated: | ||
type: string | ||
position: | ||
type: | ||
- string | ||
- "null" | ||
selfLink: | ||
type: | ||
- string | ||
- "null" | ||
completed: | ||
type: | ||
- string | ||
- "null" | ||
webViewLink: | ||
type: | ||
- string | ||
- "null" | ||
additionalProperties: true |
Oops, something went wrong.