Override table naming pattern #201
Comments
|
@kholisrag is this happening with the first run? when table created? currently there is a BigQuery issue. when connector creates the table first time and tries to write stream into it, it fails with above exception. because Bigquery needs some time to create the default stream for the new table. however when you have tables already created then this error should not happen. |
yes it is... anyway, I already have table like below
does that possible to use existing table? |
|
yes, its possible to use existing table. when the table exists the consumer will use it. and if not it will try to create it. |
I mean to change this pattern (#195 )
I only need Update:
or may I know the piece of code that do the pattern? |
|
this is currently not possible, however could be added as a new feature. |
|
thank you @ismailsimsek checking the code is that possible to use |
|
Right 👍 actually that could be used. |
|
can you give me an example? didn't find anywhere in the test code too |
|
something like this? @ismailsimsek Update: both above and below, still not working error: |
|
something like following: you could use this tool to develop the regexp https://regex101.com/ # use regexp which matches the prefix.
debezium.sink.batch.destination-regexp=^prefix\.
# use empty string as replacement
debezium.sink.batch.destination-regexp-replace= |
|
yeah I'm using https://regex101.com/ too, here https://regex101.com/r/gvzYpZ/2 Update: use below config: the table naming already replaced, but seem still not working, here is the log |
|
hmhmh when I try to use |
|
@kholisrag it seems like you are switching between |
|
but as you can see the table name is correctly set. without prefix laku6-145607:stage_postgresql_bifrost_audit_public. |
|
Okay, it seems like a mismatch schema. Checking the original table in database VM, its timestamp, not sure why its converted to an integer by debezium-server.
|
|
@kholisrag Bigquery does some level of automatic data type recognition (converting long to timestap ..etc). And this is especially different for time,timestamp types between the batch and streaming consumers. i believe this is where its coming from. i believe when you just use one consumer this should not happen. |
|
for context, we want to migrate from datastream to self-managed using this tools, without changing the existing, only want to update it. but seem like this become a problem ya...
hmhmh, I try anyway thank you! |
|
@kholisrag is there specific reason switching from data stream? I believe Data type mapping could easily be improved (especially for following config could be used to fix type mapping. (This part is still in beta) data type mapping code section is below. Type conversions and corrections could be handled here: Additionally i suggest checking debezium config to change data type values its also related to googleapis/java-bigquerystorage#1764 |
|
let me test to use: debezium.source.converters=bqdatetime
debezium.source.bqdatetime.type=io.debezium.server.converters.TemporalToISOStringConverter
cost issue @ismailsimsek |
|
i see, if you don't need real-time data. then |
|
ah I mean this GCP Datastream https://cloud.google.com/datastream/docs/overview so we want to move from GCP Datastream to self-manage @ismailsimsek , we're exploring the tools. btw, still have same issue, probably the type conversion issue, that yeah, we need to update the code to cover it. log: |
as per mentioned in #195 :
is it possible to override the table naming pattern? I only want to use
{table_name}in the table naming, does that possible @ismailsimsek?more info on log:
the config:
Thanks in advance
The text was updated successfully, but these errors were encountered: