Skip to content

Commit

Permalink
[#24789] Mention prism is the new default Go SDK runner in CHANGES.md (
Browse files Browse the repository at this point in the history
…#27816)

* mention prism in changes.md

* ws lint

* review comments

---------

Co-authored-by: lostluck <[email protected]>
  • Loading branch information
lostluck and lostluck authored Aug 2, 2023
1 parent 4d49529 commit 427e0a1
Showing 1 changed file with 12 additions and 8 deletions.
20 changes: 12 additions & 8 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,40 +54,44 @@

## Highlights

* New highly anticipated feature X added to Python SDK ([#X](https://github.com/apache/beam/issues/X)).
* New highly anticipated feature Y added to Java SDK ([#Y](https://github.com/apache/beam/issues/Y)).
* Spark 3.2.2 is used as default version for Spark runner ([#23804](https://github.com/apache/beam/issues/23804)).
* The Go SDK has a new default local runner, called Prism ([#24789](https://github.com/apache/beam/issues/24789)).

## I/Os

* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
* Python GCSIO is now implemented with GCP GCS Client instead of apitools ([#25676](https://github.com/apache/beam/issues/25676))
* Java KafkaIO now supports picking up topics via topicPattern ([#26948](https://github.com/apache/beam/pull/26948))
* Support for read from Cosmos DB Core SQL API ([#23604](https://github.com/apache/beam/issues/23604))
* Upgraded to HBase 2.5.5 for HBaseIO. (Java) ([#27711](https://github.com/apache/beam/issues/19554))

## New Features / Improvements

* X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
* The Go SDK now requires Go 1.20 to build. ([#27558](https://github.com/apache/beam/issues/27558))
* The Go SDK has a new default local runner, Prism. ([#24789](https://github.com/apache/beam/issues/24789)).
* Prism is a portable runner that executes each transform independantly, ensuring coders.
* At this point it supercedes the Go direct runner in functionality. The Go direct runner is now deprecated.
* See https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/README.md for the goals and features of Prism.

## Breaking Changes

* Legacy runner support removed from Dataflow, all pipelines must use runner v2.
* Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2.

## Deprecations

* X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)).
* The Go Direct Runner is now Deprecated. It remains available to reduce migration churn.
* Tests can be set back to the direct runner by overriding TestMain: `func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }`
* It's recommended to fix issues seen in tests using Prism, as they can also happen on any portable runner.
* Use the generic register package for your pipeline DoFns to ensure pipelines function on portable runners, like prism.
* Do not rely on closures or using package globals for DoFn configuration. They don't function on portable runners.

## Bugfixes

* Fixed DirectRunner bug in Python SDK where GroupByKey gets empty PCollection and fails when pipeline option `direct_num_workers!=1`.([#27373](https://github.com/apache/beam/pull/27373))
* Fixed BigQuery I/O bug when estimating size on queries that utilize row-level security ([#27474](https://github.com/apache/beam/pull/27474))
* Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).

## Known Issues

* ([#X](https://github.com/apache/beam/issues/X)).
* TBD


# [2.49.0] - 2023-07-17
Expand Down

0 comments on commit 427e0a1

Please sign in to comment.