Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Managed Iceberg] Add support for TIMESTAMP, TIME, and DATE types #32688

Merged
merged 15 commits into from
Oct 10, 2024
Merged

[Managed Iceberg] Add support for TIMESTAMP, TIME, and DATE types #32688

merged 15 commits into from
Oct 10, 2024

Conversation

ahmedabu98
Copy link
Contributor

Fixes #32680

@ahmedabu98 ahmedabu98 mentioned this pull request Oct 7, 2024
Closed
17 tasks
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 8, 2024

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@ahmedabu98
Copy link
Contributor Author

assign set of reviewers

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 8, 2024

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.
R: @Abacn for label build.
R: @shunping for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@ahmedabu98
Copy link
Contributor Author

Thanks for taking an initial look @DanielMorales9. This PR is ready for another review

@ahmedabu98
Copy link
Contributor Author

@robertwb @Abacn @shunping this could also use some eyes from Beam devs

@ahmedabu98
Copy link
Contributor Author

ahmedabu98 commented Oct 9, 2024 •
edited
Loading

Beam (FieldType.DATETIME) and Iceberg (timestamptz) handle timestamps with timezone in a similar way -- they resolve to UTC and drop the timezone information.

I added support for String timestamps to be parsed to OffsetDateTime. But I wonder if it makes sense to allow other types as well (LocalDateTime, joda DateTime, long) and resolve all these types to OffsetDateTime at UTC. This will allow us to have a more unified approach to timestamps - and Iceberg always resolves to UTC anyways so there's no harm in doing it from our side

@ahmedabu98
Copy link
Contributor Author

Thanks for the LGTM -- will merge when tests pass

@ahmedabu98 ahmedabu98 merged commit 9ceb14e into apache:master Oct 10, 2024
22 checks passed
@ahmedabu98 ahmedabu98 mentioned this pull request Oct 11, 2024
Open
17 tasks
reeba212 pushed a commit to reeba212/beam that referenced this pull request Dec 4, 2024
@ahmedabu98 @reeba212
[Managed Iceberg] Add support for TIMESTAMP, TIME, and DATE types (ap…
0e90f5a 
…ache#32688)

* support timestamp, time, date types

* add to changes md

* always write java time LocalDateTime for iceberg TIMESTAMP

* update java doc

* add timezone support with Strings

* clean up; reading iceberg timestamptz will return sqltype.datetime

* support string, long, sql.datetime, and datetime; timestamp returns sql.datetime and timestamptTZ returns datetime
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DanielMorales9 DanielMorales9 DanielMorales9 left review comments

Assignees
No one assigned
Labels
build io java Next Action: Reviewers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: IcebergIO cannot write Timestamp columns
2 participants
@ahmedabu98 @DanielMorales9