Skip to content
This repository has been archived by the owner on Nov 11, 2022. It is now read-only.

Commit

Permalink
Merge pull request #504 from rangadi/kafka_release_notes
Browse files Browse the repository at this point in the history
Increase KafkaIO version to 0.2.0
  • Loading branch information
dhalperi authored Jan 18, 2017
2 parents 8fd6661 + 40d174f commit 2e57ab1
Show file tree
Hide file tree
Showing 2 changed files with 40 additions and 1 deletion.
39 changes: 39 additions & 0 deletions contrib/kafka/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# KafkaIO : Dataflow Unbounded Source and Sink for Kafka Topics

KafkaIO provides unbounded source and sink for [Kafka](http://kafka.apache.org/)
topics. Kafka versions 0.9 and above are supported.

## Basic Usage

* Read from a topic with 8 byte long keys and string values:
```java
PCollection<KV<Long, String>> kafkaRecords =
pipeline
.apply(KafkaIO.read()
.withBootstrapServers("broker_1:9092,broker_2:9092")
.withTopics(ImmutableList.of("topic_a"))
.withKeyCoder(BigEndianLongCoder.of())
.withValueCoder(StringUtf8Coder.of())
.withoutMetadata()
);
```

* Write the same PCollection to a Kafka topic:
```java
kafkaRecords.apply(KafkaIO.write()
.withBootstrapServers("broker_1:9092,broker_2:9092")
.withTopic("results")
.withKeyCoder(BigEndianLongCoder.of())
.withValueCoder(StringUtf8Coder.of())
);
```

Please see JavaDoc for KafkaIO in
[KafkaIO.java](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/contrib/kafka/src/main/java/com/google/cloud/dataflow/contrib/kafka/KafkaIO.java#L100)
for complete documentation and a more descriptive usage example.

## Release Notes
* **0.2.0** : Assign one split for each of the Kafka topic partitions. This makes Dataflow
[Update](https://cloud.google.com/dataflow/pipelines/updating-a-pipeline)
from previous version incompatible.
* **0.1.0** : KafkaIO with support for Unbounded Source and Sink.
2 changes: 1 addition & 1 deletion contrib/kafka/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<artifactId>google-cloud-dataflow-java-contrib-kafka</artifactId>
<name>Google Cloud Dataflow Kafka IO Library</name>
<description>Library to read Kafka topics.</description>
<version>0.1.0-SNAPSHOT</version>
<version>0.2.0-SNAPSHOT</version>

<properties>
<dataflow.version>[1.6.0, 1.99)</dataflow.version>
Expand Down

0 comments on commit 2e57ab1

Please sign in to comment.