v0.8.1

Major Changes

• Ability to configure different base href for Numaflow UI server.
• Restart the Kafka source pod when there's a server side error.
• Fixed the message size limitation issue (only messages with size < 32K were allowed).

Breaking Changes and Upgrading Guide

• ISBSVC with Redis 6.0.6 is not supported any more, using 7.0.11. Anyone who is using old version, please delete the old object, update the version and recreate.

• The buffer and bucket definitions are revised, therefore we updated the buffer and bucket implementation, which requires to recreate the pipeline during upgrade (1. Delete the pipelines; 2. Upgrade; 3. Recreate the pipelines). Meanwhile, we deprecated following properties:

  1. Deprecated spec.edges[].parallelism, use spec.vertices[].partitions.
  2. Deprecated spec.edges[].limits.*, use spec.vertices[].limits.*, * includes bufferMaxLength and bufferUsageLimit.

Pull Requests

v0.8.0...v0.8.1

Contributors

@KeranYang
@vigith
@yhl25
@juliev0
@chromevoid
@dpadhiar
@veds-g
@whynowy
@kohlisid
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.8.1/install.yaml

v0.8.0

Check out our release blog.

Major Changes

• Ability to assign event time, as well as filtering and transformations at the source vertex using source transformers
• Idle watermark propagation now does watermark progression in idle edges reducing close-of-book latency
• Message delivery count is now being tracked and is exposed to user-defined code
• Support Redis Streams Source
• Performance improvements
  • Message SerDe has been optimized for better encoding/decoding
  • WAL write/read CPU performance was boosted by 20%, and disk space optimization by close to 40%
• Controller to support high availability
• Kill switch feature - the ability to drop a message on demand when the buffer is full
• UX improvements
  • Ability to do namespace scoped installation
  • Switched to namespace search on the main page
  • Be able to show reduce vertex and edge information
• SASL support for both Kafka source and sink
• Late data processing using allowedLateness

Breaking Changes

• Introduced tags for conditional forwarding, removed keyIn, see the detail at here
• SDK changes
  • Please move to the latest 0.4.x release of each SDK
  • Python
    • Major refactor on server startup, please review the README
    • Added Async Map and Async Sink
    • Map has multi-process support via SO_REUSEPORT
    • Removed Python 3.8 support and added support for 3.11

Pull Requests

v0.7.0...v0.8.0

Contributors

@KeranYang
@vigith
@yhl25
@ashwinidulams
@juliev0
@chromevoid
@dpadhiar
@veds-g
@sandangel
@whynowy
@kohlisid
@xdevxy
@alexanghh

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.8.0/install.yaml

v0.8.0-rc1

Major Changes

• Ability to assign event time, as well as filtering and transformations at the source vertex using source transformers
• Idle watermark propagation now does watermark progression in idle edges reducing close-of-book latency
• Message delivery count is now being tracked and is exposed to user-defined code
• Support Redis Streams Source
• Performance improvements
  • Message SerDe has been optimized for better encoding/decoding
  • WAL write/read CPU performance was boosted by 20%, and disk space optimization by close to 40%
• Controller to support high availability
• Kill switch feature - the ability to drop a message on demand when the buffer is full
• UX improvements
  • Ability to do namespace scoped installation
  • Switched to namespace search on the main page
  • Be able to show reduce vertex and edge information
• SASL support for both Kafka source and sink

Breaking Changes

• Introduced tags for conditional forwarding, removed keyIn, see the detail at here
• SDK changes
  • Please move to the latest 0.4.x release of each SDK
  • Python
    • Major refactor on server startup, please review the README
    • Added Async Map and Async Sink
    • Map has multi-process support via SO_REUSEPORT
    • Removed Python 3.8 support and added support for 3.11

Pull Requests

v0.7.0...v0.8.0-rc1

Contributors

@KeranYang
@vigith
@yhl25
@ashwinidulams
@juliev0
@chromevoid
@dpadhiar
@veds-g
@sandangel
@whynowy
@kohlisid
@xdevxy
@alexanghh

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.8.0-rc1/install.yaml

v0.7.3

Major Changes

• Kill Switch to avoid ISB overruns (#524)
• UX Improvements
• Performance improvements in WAL SerDe
• SASL support for Kafka Source
• Idle Watermark Propagation

Pull Requests

v0.7.2...v0.7.3

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.3/install.yaml

v0.7.2

Major Changes

• Continuous performance improvement and bug fixes for Reduce operation, switched to bidirectional streaming.
• Watermark propagation improvement to reduce delay
• UI changes including reduce reduce vertex support, and enabling namespace scoped installation.
• Controll HA support
• targetBufferUsage for vertex autoscaling is deprecated, use targetBufferAvailability instead, will be removed in v0.9.

Pull Requests

v0.7.1...v0.7.2

Contributors

@KeranYang
@yhl25
@vigith
@chromevoid
@whynowy
@dpadhiar
@ashwinidulams
@kohlisid
@veds-g

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.2/install.yaml

v0.7.1

Major Changes

• Major performance improvement for Reduce operation
  • de-multiplexing gRPC streams
  • asynchronous development pattern is recommended for all SDKs

Breaking Changes

• gRPC streams are per window per slot, will work only with latest 0.3.x SDK versions
• only asyncio is supported for Python
• Java uses Akka

Pull Requests

v0.7.0...v0.7.1

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.1/install.yaml

v0.7.0

Major Changes

• Reduce with Windowing is now possible
  • Fixed
  • Sliding

You can read about the new Windowing and Reduce feature in our latest blog.

Breaking Changes

• UDSink is now a streaming RPC call (no longer unary)
  • Please use SDK version 0.3.x

Pull Requests

v0.6.5...v0.7.0

Contributors

@DevilAeron
@chromevoid
@dseapy
@whynowy
@KeranYang
@yhl25
@xdevxy
@vigith
@ashwinidulams
@si3nloong
@dratler
@edlee2121

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.7.0/install.yaml

v0.7.0-rc1

Major Changes

• Windowing is now possible
  • Fixed
  • Sliding
• Aggregation feature (Reduce)

Breaking Changes

• UDSink is now a streaming RPC call (no longer unary)
  • Please use SDK version 0.3.x

Pull Requests

• feat: reduce metrics. Closes #313 (#414)
• feat: udsink grpc stream (#421)
• test: sliding window e2e test (#427)
• chore: validate if pipeline or vertex name too long (#428)
• chore(doc): scope UDF under a dir (#426)
• chore: refactor reduce ctx (#419)
• chore: update PBQ WAL metrics (#422)
• feat: sliding window. closes #339 (#354)
• refactor: nats/jetstream testing (#418)
• feat: nats as source (#411)
• docs: updated CHANGELOG.md (#415)
• chore: Windower interface refactor (#409)
• fix: adding lock while discovering partitions, Closes #412 (#413)
• fix(test): e2e-api-pod can not start on M1 mac (#410)
• chore: e2e tests for reduce. Closes #338 (#386)
• fix: getWatermark to return-1 if any processor returns -1 (#402)
• chore: add pipeline and vertex name validation. Closes #342 (#398)
• chore: default logs to user defined containers if there are. Close #283 (#397)
• fix: e2e testing for PBQ WAL with reduce pipeline (#393)
• chore: pass vertex/pipeline name to pbq wal and add a grafana dashboard template for wal metrics. Closes #365 (#385)
• feat: add Grafana instruction and a dashboard template. Closes #287 (#381)
• chore: reduce max msgs for jetstream config (#395)
• chore:non keyed stream support for reduce (#394)
• chore: fix broken link (#391)
• chore: adding DeleteStore method to store provider interface (#390)
• fix: unit tests for replay. Closes #373 (#377)
• chore: switch to sync publish mode for jetstream isb (#388)
• chore(docs): update docs (#380)
• chore: make tickgen ID unique (#379)
• chore: concurrent acks inside readloop (#376)
• chore: add error metrics for WAL (#374)
• fix: best effort processing during SIGTERM. Closes #371 (#372)
• chore: clean up unused code (#370)
• feat(wal): First pass to implement WAL and hook to PBQ store. (#344)
• feat: watermark otwatcher enhancement (#364)
• refactor(docs): group docs in categories (#362)
• chore: vertex controller stop watching pod create events (#361)
• chore(deps): bump loader-utils from 2.0.3 to 2.0.4 in /ui (#356)
• fix(controller): vertex nil check for edge listing. Fixes #352 (#353)
• chore: minor window refactor (#347)
• fix: data race in pbq manager. Closes #348 (#349)
• Chore: Windower interface. closes #234 (#340)
• chore: fix broken development doc and remove incorrect make cmds (#346)
• feat: add minikube, kind and podman support (#206)
• refactor: Close watermark fetcher and publisher correctly (#336)
• passing window information inside the context (#341)
• feat: timestamp in UI to display milliseconds. closes #280 (#337)
• Simple reduce pipeline. Fixes #289 (#317)
• feat: add blackhole sink. Closes #329 (#330)
• fix: move watermark based on the head of the read batch (#332)
• feat: configurable jetstream storage (#328)
• feat: support adding sidecars in vertex pods. Closes #323 (#325)
• feat: populate watermark settings to vertex spec. Closes #320 (#321)
• chore: use github action services and upgrade Jetstream (#322)
• doc: add few use cases (#318)
• chore: set GOMEMLIMIT and increase graceshutdownperiod (#319)
• Chore: run in sdks-e2e tests, python-udsink log check before go-udsink (#315)
• fix: jetstream build watermark progressors bug (#316)
• feat: update watermark offset bucket implementation (#307)
• chore: upgrade xmldom to address security concerns (#309)
• feat: shuffling support (#306)
• chore: Add boilerplate headers (#308)
• feat: customize init-container resources. Closes #303 (#304)
• feat: watermark - remove non-share OT bucket option (#302)
• feat: customization for batch jobs. Closes #259 (#300)
• refactor: abstract pod template (#296)
• feat: customization for daemon deployment. Closes #223 (#290)
• chore: use milliseconds in HTTP event-time header (#295)
• feat: add pvc support for reduce vertex PBQ (#292)
• fix(doc): hyperlink for security doc (#288)
• feat: support adding init containers to vertices. Closes #284 (#285)
• docs: fix spelling in docs and examples (#286)
• fix: retry when getting EOF error at E2E test (#281)
• feat: Watermark millisecond. Fixes #201 (#278)
• feat: add pipeline node counts and age to printcolumn. Closes #267 (#282)
• feat: introduce reduce UDF. Closes #246 (#262)
• feat: add pandoc to required tools development doc. Closes #276 (#277)
• feat: add isbsvc type and age to printcolumn. Closes #268 (#275)
• fix: watermark consumer fix (#273)
• refactor: generalize watermark fetching as an interface of ISB service. Fixes #252 (#263)
• fix: set default property values for minimal CRD installation (#264)
• fix: validate only one isbsvc implementation type is defined. Fixes #269 (#271)
• fix: main branch make build failure: math.MaxInt64 for int type (#265)
• fix: nil pointer deref when running example with minimal CRD. Fixes #260 (#261)
• fix: retry only the failed offsets (#255)
• docs: update ISB redis doc (#258)
• fix: re-enable build constraint on processor manager test. Fixes #256 (#257)
• fix: container resource for jetstream isbsvc. Fixes #253 (#254)
• fix: update vertex watermark fetching logic. Fixes: #134 (#245)
• chore: refactor to include partition.ID for reducer (#243)
• chore: introduce simple int offset (#250)
• fix: watermark watcher leak (#242)
• fix(docs): fix a typo (#241)
• feat: Support running UX server with namespace scope. Fixes #248 (#249)
• test: reduce test with inmemory watermark store (#239)
• fix(manifests): Include ServiceAccount in namespace scoped install (#240)
• fix: Watermark close fix and removed the nil check (#238)
• fix: skip publishing watermarks to unexpected vertices. Fixes #235 (#236)
• fix: update default watermark to -1. Fixes #133 (#218)
• feat: support disabling TLS and changing port for UX server (#228)
• feat: reducer for stream aggregation without fault tolerance (#208)
• feat: in-memory watermark store for better testing (#216)
• Add USERS.md (#221)
• fix(watermark): generator should not publish wm for every message (#217)
• chore: reference the Payload directly when writing to Redis ISB to avoid extra marshaling (#213)

Contributors

@DevilAeron
@chromevoid
@dseapy
@whynowy
@KeranYang
@yhl25
@xdevxy
@vigith
@ashwinidulams
@si3nloong
@dratler
@edlee2121

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.7.0-rc1/config/install.yaml

v0.6.5

Major Changes

• Watermark related bug fixes
• Fine tune JetStream retention policy
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• chore: Windower interface refactor (#409)
• fix: adding lock while discovering partitions, Closes #412 (#413)
• fix(test): e2e-api-pod can not start on M1 mac (#410)
• chore: e2e tests for reduce. Closes #338 (#386)
• fix: getWatermark to return-1 if any processor returns -1 (#402)
• chore: add pipeline and vertex name validation. Closes #342 (#398)
• chore: default logs to user defined containers if there are. Close #283 (#397)
• fix: e2e testing for PBQ WAL with reduce pipeline (#393)
• chore: pass vertex/pipeline name to pbq wal and add a grafana dashboard template for wal metrics. Closes #365 (#385)
• feat: add Grafana instruction and a dashboard template. Closes #287 (#381)
• chore: reduce max msgs for jetstream config (#395)
• chore:non keyed stream support for reduce (#394)
• chore: fix broken link (#391)
• chore: adding DeleteStore method to store provider interface (#390)
• fix: unit tests for replay. Closes #373 (#377)

Contributors

@KeranYang
@whynowy
@yhl25
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.5/config/install.yaml

v0.6.4

Major Changes

• Switched JetStream ISB writing from async to concurrent sync
• Pre-released features for v0.7

Check out v0.6 release blog.

Pull Requests

• chore: switch to sync publish mode for jetstream isb (#388)
• chore(docs): update docs (#380)
• chore: make tickgen ID unique (#379)
• chore: concurrent acks inside readloop (#376)
• chore: add error metrics for WAL (#374)
• fix: best effort processing during SIGTERM. Closes #371 (#372)
• chore: clean up unused code (#370)
• feat(wal): First pass to implement WAL and hook to PBQ store. (#344)

Contributors

@vigith
@whynowy
@yhl25
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/v0.6.4/config/install.yaml