From 427e0a1c852a798fd2b1105672efdcfcdc33cd07 Mon Sep 17 00:00:00 2001 From: Robert Burke Date: Wed, 2 Aug 2023 13:57:45 -0700 Subject: [PATCH] [#24789] Mention prism is the new default Go SDK runner in CHANGES.md (#27816) * mention prism in changes.md * ws lint * review comments --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> --- CHANGES.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index c6e756ee9646..62e090a87324 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -54,13 +54,11 @@ ## Highlights -* New highly anticipated feature X added to Python SDK ([#X](https://github.com/apache/beam/issues/X)). -* New highly anticipated feature Y added to Java SDK ([#Y](https://github.com/apache/beam/issues/Y)). * Spark 3.2.2 is used as default version for Spark runner ([#23804](https://github.com/apache/beam/issues/23804)). +* The Go SDK has a new default local runner, called Prism ([#24789](https://github.com/apache/beam/issues/24789)). ## I/Os -* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)). * Python GCSIO is now implemented with GCP GCS Client instead of apitools ([#25676](https://github.com/apache/beam/issues/25676)) * Java KafkaIO now supports picking up topics via topicPattern ([#26948](https://github.com/apache/beam/pull/26948)) * Support for read from Cosmos DB Core SQL API ([#23604](https://github.com/apache/beam/issues/23604)) @@ -68,26 +66,32 @@ ## New Features / Improvements -* X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)). * The Go SDK now requires Go 1.20 to build. ([#27558](https://github.com/apache/beam/issues/27558)) +* The Go SDK has a new default local runner, Prism. ([#24789](https://github.com/apache/beam/issues/24789)). + * Prism is a portable runner that executes each transform independantly, ensuring coders. + * At this point it supercedes the Go direct runner in functionality. The Go direct runner is now deprecated. + * See https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/README.md for the goals and features of Prism. ## Breaking Changes -* Legacy runner support removed from Dataflow, all pipelines must use runner v2. +* Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2. ## Deprecations -* X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)). +* The Go Direct Runner is now Deprecated. It remains available to reduce migration churn. + * Tests can be set back to the direct runner by overriding TestMain: `func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }` + * It's recommended to fix issues seen in tests using Prism, as they can also happen on any portable runner. + * Use the generic register package for your pipeline DoFns to ensure pipelines function on portable runners, like prism. + * Do not rely on closures or using package globals for DoFn configuration. They don't function on portable runners. ## Bugfixes * Fixed DirectRunner bug in Python SDK where GroupByKey gets empty PCollection and fails when pipeline option `direct_num_workers!=1`.([#27373](https://github.com/apache/beam/pull/27373)) * Fixed BigQuery I/O bug when estimating size on queries that utilize row-level security ([#27474](https://github.com/apache/beam/pull/27474)) -* Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)). ## Known Issues -* ([#X](https://github.com/apache/beam/issues/X)). +* TBD # [2.49.0] - 2023-07-17