README.md

TypeStream

Why TypeStream? · Getting started · How to contribute · Code of conduct · License

TypeStream is an abstraction layer on top of Kafka that allows you to write and run typed data pipelines with a minimal, familiar syntax.

Why TypeStream?

Building streaming data pipelines on top of Kafka comes with some fixed costs. You have to write an app, test it, then deploy and manage it in production. Even for the simplest pipelines, this can be a lot of work.

With TypeStream you can write powerful, typed data pipelines the way you'd write a simple UNIX pipeline in your terminal. For example, imagine you'd like to filter a "books" topic. With TypeStream, it's a one liner:

$ typestream
> grep /dev/kafka/local/topics/books the > /dev/kafka/local/topics/books_with_the

TypeStream will take care of type-checking your pipeline and then run it for you. Here's how it looks like in action:

Another common use case that requires a lot of boilerplate is to enrich a topic with data from an external source. For example, you might have a topic with an ip address field and you may want to enrich it with country information. With TypeStream, you can do it (again!) in a single line:

$ typestream
> cat /dev/kafka/local/topics/page_views | enrich { view -> http "https://api.country.is/#{$view.ip_address}" | cut .country } > /dev/kafka/local/topics/page_views_with_country

Here's how enriching looks like in action:

As you can see from the previous command, in the spirit of UNIX, we used cut to extract the country field from the response. Here's the kick, you can use cut (and many other Unix commands) on streams as well:

$ typestream
> cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles

Here's how cutting looks like in action:

Another problem that TypeStream solves is preventing you from writing faulty pipelines by type-checking them before executing them. Here's type checking in action:

Deploying a pipeline to production is as simple as running:

typestream run 'cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles'

See it in action here:

If you'd like to learn more about TypeStream, check out the official documentation.

Getting started

If you use Homebrew:

brew install typestreamio/tap/typestream
$ typestream --version

if you see something like this:

typestream version 2023.08.31+3 42f7762daac1872416bebab7a34d0b79a838d40a (2023-09-02 09:20:52)

then you're good to go! You can now run:

typestream local start
typestream local seed

to start a local TypeStream server and seed it with some sample data. Now you're ready to start writing your own pipelines!

Check out our documentation to learn more about TypeStream.

How to contribute

We love every form of contribution! Good entry points to the project are:

• Our contributing guidelines document.
• Issues with the tag gardening.
• Issues with the tag good first patch.

If you're not sure where to start, open a new issue or hop on to our discord server and we'll gladly help you get started.

Code of Conduct

You are expected to follow our code of conduct when interacting with the project via issues, pull requests, or in any other form. Many thanks to the awesome contributor covenant initiative!

License

Apache 2.0