Skip to content
This repository has been archived by the owner on Nov 11, 2022. It is now read-only.

Version 0.4.20150727

Pre-release
Pre-release
Compare
Choose a tag to compare
@davorbonaci davorbonaci released this 28 Jul 17:18
· 1497 commits to master since this release
  • Removed the requirement to explicitly set --project if Google Cloud SDK has the default project configuration set.
  • Added support for creating BigQuery sources from a query.
  • Added support for custom unbounded sources in the DirectPipelineRunner and DataflowPipelineRunner. See UnboundedSource for details.
  • Removed unnecessary ExecutionContext argument in BoundedSource.createReader and related methods.
  • Changed BoundedReader.splitAtFraction to require thread-safety (i.e. safe to call asynchronously with advance or start). Added RangeTracker to help implement thread-safe readers. Users are heavily encouraged to use the class rather than implementing an ad-hoc solution.
  • Modified Combine transforms by lifting them into (and above) the GroupByKey resulting in better performance.
  • Modified triggers such that after a GroupByKey, the system will switch to a "Continuation Trigger", which attempts to preserve the original intention regarding handling of speculative and late triggerings instead of returning to the default trigger.
  • Added WindowFn.getOutputTimestamp and changed GroupByKey behavior to allow incomplete overlapping windows to not hold up progress of earlier, completed windows.
  • Changed triggering behavior so that empty panes are produced if they are the first pane after the watermark (ON_TIME) or the final pane.
  • Removed the Window.Trigger intermediate builder class.
  • Added validation that allowed lateness is specified on the Window PTransform when a trigger is specified.
  • Re-enabled verification of GroupByKey usage. Specifically, the key must have a deterministic coder and using GroupByKey with an unbounded PCollection requires windowing or triggers.
  • Changed PTransform names so that they may no longer contain the = or ; characters.