Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Managed Iceberg dynamic destinations #32565

Merged
merged 9 commits into from
Sep 30, 2024

Conversation

ahmedabu98
Copy link
Contributor

Fixes #32365

This PR adds support for writing to dynamic Iceberg destinations, meaning that one transform instance can write to multiple Iceberg tables.

IcebergIO is offered as a Managed transform, which makes it a portable transform. Instead of offering a function of some sort, users can specify a String template and we will substitute in the appropriate record field values to determine the destination (see RowStringInterpolator for more details and examples)

For more information on portable dynamic destinations, see #32365

@ahmedabu98 ahmedabu98 changed the title iceberg dynamic destinations Managed Iceberg dynamic destinations Sep 25, 2024
@ahmedabu98
Copy link
Contributor Author

assign set of reviewers

Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label java.
R: @Abacn for label build.
R: @shunping for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Copy link
Contributor

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

LGTM

@chamikaramj
Copy link
Contributor

Thanks. LGTM.

@ahmedabu98 ahmedabu98 merged commit deb4b4a into apache:master Sep 30, 2024
30 of 31 checks passed
reeba212 pushed a commit to reeba212/beam that referenced this pull request Dec 4, 2024
* iceberg dynamic destinations

* update CHANGES

* spotless

* fix null error

* fix some tests

* fix some tests

* better row filter error message

* clarify keep-only-drop mutual exclusivity

* use constants for field names
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task]: Add utilities to easily implement portable dynamic destinations
2 participants