This repository has been archived by the owner on Nov 11, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 322
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #504 from rangadi/kafka_release_notes
Increase KafkaIO version to 0.2.0
- Loading branch information
Showing
2 changed files
with
40 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
# KafkaIO : Dataflow Unbounded Source and Sink for Kafka Topics | ||
|
||
KafkaIO provides unbounded source and sink for [Kafka](http://kafka.apache.org/) | ||
topics. Kafka versions 0.9 and above are supported. | ||
|
||
## Basic Usage | ||
|
||
* Read from a topic with 8 byte long keys and string values: | ||
```java | ||
PCollection<KV<Long, String>> kafkaRecords = | ||
pipeline | ||
.apply(KafkaIO.read() | ||
.withBootstrapServers("broker_1:9092,broker_2:9092") | ||
.withTopics(ImmutableList.of("topic_a")) | ||
.withKeyCoder(BigEndianLongCoder.of()) | ||
.withValueCoder(StringUtf8Coder.of()) | ||
.withoutMetadata() | ||
); | ||
``` | ||
|
||
* Write the same PCollection to a Kafka topic: | ||
```java | ||
kafkaRecords.apply(KafkaIO.write() | ||
.withBootstrapServers("broker_1:9092,broker_2:9092") | ||
.withTopic("results") | ||
.withKeyCoder(BigEndianLongCoder.of()) | ||
.withValueCoder(StringUtf8Coder.of()) | ||
); | ||
``` | ||
|
||
Please see JavaDoc for KafkaIO in | ||
[KafkaIO.java](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/contrib/kafka/src/main/java/com/google/cloud/dataflow/contrib/kafka/KafkaIO.java#L100) | ||
for complete documentation and a more descriptive usage example. | ||
|
||
## Release Notes | ||
* **0.2.0** : Assign one split for each of the Kafka topic partitions. This makes Dataflow | ||
[Update](https://cloud.google.com/dataflow/pipelines/updating-a-pipeline) | ||
from previous version incompatible. | ||
* **0.1.0** : KafkaIO with support for Unbounded Source and Sink. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters