doc: Add documentation+FAQs for Parquet DataSource #153
Conversation
…' into feat/issue#109-add-parquet-source-documentation
|
Added the configurations + some common gotchas for setting up configurations. Also, some FAQs and how to create the stream config for parquet. I haven’t touched the diagrams yet, as they will require some effort to be redrawn since the original resources are missing. However, have changed the description in relevent places. |
docs/docs/guides/create_dagger.md
Outdated
|
|
||
| ``` | ||
|
|
||
| or data partitioned, such as: |
There was a problem hiding this comment.
small typo here, I believe we meant date partitioned here.
There was a problem hiding this comment.
Fixed via commit 31f5ec3
…' into feat/issue#109-add-parquet-source-documentation
docs/docs/concepts/architecture.md
Outdated
| - A Stream defines a logical grouping of a data source and its associated [`protobuf`](https://developers.google.com/protocol-buffers) | ||
| schema. All data produced by a source follows the protobuf schema. The source can be a bounded one such as `KAFKA_SOURCE` or `KAFKA_CONSUMER` | ||
| in which case, a single stream can consume from one or more topics all sharing the same schema. Otherwise, the source | ||
| can be an unbounded one such as `PARQUET_SOURCE` in which case, one or more parquet files as provided are consumed in a single stream. |
There was a problem hiding this comment.
Kafka is unbounded and parquet is bounded data source. Fixing this as well
There was a problem hiding this comment.
Fixed via commit 91a73a5
docs/docs/concepts/basics.md
Outdated
| A Stream defines a logical grouping of a data source and its associated [`protobuf`](https://developers.google.com/protocol-buffers) | ||
| schema. All data produced by a source follows the protobuf schema. The source can be a bounded one such as `KAFKA_SOURCE` or `KAFKA_CONSUMER` | ||
| in which case, a single stream can consume from one or more topics all sharing the same schema. Otherwise, the source | ||
| can be an unbounded one such as `PARQUET_SOURCE` in which case, one or more parquet files as provided are consumed in a single stream. |
There was a problem hiding this comment.
Same as https://github.com/odpf/dagger/pull/153/files#r890081887. Will fix this
There was a problem hiding this comment.
Fixed via commit 91a73a5
PR for #109