Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support logical types when translating the portable row coder representation #19715

Closed
damccorm opened this issue Jun 4, 2022 · 1 comment · Fixed by #28382
Closed

Support logical types when translating the portable row coder representation #19715

damccorm opened this issue Jun 4, 2022 · 1 comment · Fixed by #28382

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 4, 2022

Originally SchemaCoder and RowCoder relied on including serialized java classes and functions to support logical types, but the portable schema representation only includes a URN for a logical type. We will need to be able to re-construct logical types given just a URN, presumably by creating some sort of logical type registry indexed by URN.

Imported from Jira BEAM-7855. Original Jira may contain additional context.
Reported by: bhulette.

@Abacn
Copy link
Contributor

Abacn commented Aug 30, 2022

From the TODO mark in source:

Any known but yet standardized logical types (those in STANDARD_LOGICAL_TYPES) will assign the same urn "beam:logical_type:javasdk:v1". These include all jdbc logical types and also many defined in org.apache.beam.sdk.schemas.logicaltypes package. While user can register this URN and implement with there own class in a cross-language pipeline, (e.g. https://stackoverflow.com/questions/68758361/error-beamlogical-typejavasdkv1-while-using-apache-beam-io-jdbc-readfromjdbc), they can do at most one logical type because they share the same URN. Would be preferrable to encode the identifier in URN as well, for example, in the example stackoverflow case, The involved logical type from Java side is org.apache.beam.sdk.io.jdbc.LogicalTypes.VariableLengthString with identifier 'VARCHAR', It would then have URN "beam:logical_type:javasdk_varchar::v1" or "beam:logical_type:javasdk_variablelengthstring::v1" (or "beam:logical_type:javasdk:varchar::v1" or "beam:logical_type:javasdk:variablelengthstring::v1")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants