Fixing Debezium integration tests #11154

dlg99 · 2021-06-29T21:12:29Z

Fixes #11099

Motivation

Integration test for debezium connector to tun on CI

Modifications

Debezium integration test for postgres: added check for flush lsn updates

Added pulsar-io tests to the CI

Verifying this change

Ran locally:

mvn package -DintegrationTests -f tests/docker-images/pom.xml

mvn test -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-io-sources.xml -DintegrationTests -Dtest=org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumSourcesTest

Make sure that the change passes the CI checks.

This change fixes existing tests.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

NO

Documentation

Does this pull request introduce a new feature? NO

eolivelli

Very good.

@lhotari @codelipenghui @congbobo184 @sijie PTAL

lhotari

Good work @dlg99 !

I was wondering how the tests weren't run in CI. I tracked it back to #10140 when tests were refactored. We missed the CI part for Pulsar IO in that PR. Thanks for fixing it as part of this PR.

dlg99 · 2021-07-07T18:12:05Z

/pulsarbot run-failure-checks

dlg99 · 2021-07-07T18:12:48Z

Pulsar IO integration test failed after rebase, likely the same reason as #11204 (comment)

dlg99 · 2021-07-07T21:11:41Z

I confirmed that tests pass after revert of PIP-85; @nlu90 is looking at the options to fix this without revert.

eolivelli · 2021-07-15T05:59:17Z

Failed test
testDebeziumMongoDbSource(org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumSourcesTest) Time elapsed: 33.666 s <<< FAILURE!
java.lang.AssertionError: expected [1] but found [0]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.testng.Assert.assertEquals

dlg99 · 2021-07-15T21:04:30Z

@nlu90 @eolivelli situation with this PR so far:
I rebased it on latest master and "CI - Integration - Pulsar-IO Sinks and Sources" failed.
Running testDebeziumMySqlSourceJson locally I can repro failures, rootcause TBD but the only suspicious thing that I see in the function log is

[pulsar-client-io-1-2] ERROR org.apache.pulsar.io.kafka.connect.AbstractKafkaConnectSource - Timed out waiting to flush org.apache.pulsar.io.kafka.connect.KafkaConnectSource$KafkaSourceRecord@72495bb1 offsets to storage

I reverted both PRs #11251 and #11056 locally; rebuilt/re-run the test and the test passed, "Timed out waiting to flush" is not logged in the function log

dlg99 · 2021-07-16T01:08:25Z

tests do pass if I create a new pulsar client in the PulsarOffsetBackingStore's ctor instead of the one passed there (I hardcoded the url in the test).

There is still something fishy with passing the client via source context and, ironically, I am missing on a context of what's going on there. Feels like some kind of deadlock at the PulsarOffsetStore.set() where it does producer.flushAsync().

I suggest we merge this PR, and @nlu90 continues work on the client in the SourceContext with the test enabled on the CI.

nlu90 · 2021-07-19T22:19:37Z

@dlg99 Let me know when you merge this PR, I will continue look into the fix.

dlg99 · 2021-07-19T22:39:05Z

@nlu90 I don't have permissions to merge; I hope @codelipenghui / @sijie / @merlimat / @eolivelli or other committer can merge.

eolivelli · 2021-07-20T08:17:21Z

@nlu90 are you "approving" this patch ?

eolivelli · 2021-07-20T08:19:20Z

I am fine to merge this patch as soon as CI is green

dlg99 · 2021-07-20T15:21:21Z

@eolivelli

I am fine to merge this patch as soon as CI is green

I don't know what's going on with "CI - Unit - Brokers - Broker Group 1 / unit-tests" but it is not related.
"CI - Integration - Pulsar-IO Sinks and Sources" won't be green unless the problem with debezium/pulsar client in the source context is fixed.
The point of merging this to enable integration tests on the PR and break the circular dependency where @nlu90 does the fixes but I detect the problems by rebasing this PR.

sijie · 2021-07-23T20:44:43Z

I think the agreement is to let Neng work on a proper fix based on Andrey's PR.

nlu90 · 2021-07-23T21:57:38Z

I pushed a fix for the debezium connector in #11435

And the integration test passed on my local laptop:

➜  sn-pulsar git:(neng/kafka-source-async-ack) ✗ mvn test -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-io-sources.xml -DintegrationTests -Dtest=org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumSourcesTest
...
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Pulsar :: Tests 2.9.0-SNAPSHOT:
[INFO]
[INFO] Apache Pulsar :: Tests ............................. SUCCESS [  1.147 s]
[INFO] Apache Pulsar :: Tests :: Integration .............. SUCCESS [06:12 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:13 min
[INFO] Finished at: 2021-07-23T11:05:54-07:00
[INFO] ------------------------------------------------------------------------

dlg99 · 2021-07-23T22:01:54Z

@nlu90 I rebased the PR on top of recent master, the IO tests still failed.
The error looks different:

2021-07-23 21:25:17,576 - INFO  - [docker-java-stream-1663661242:DockerUtils$2@211] - DOCKER.exec(PulsarDebeziumSourcesTest-abzgk-pulsar-functions-worker-process-chgkv-1:/pulsar/bin/pulsar-admin source delete --tenant public --namespace default --name test-source-debezium-mysqlavro-PROCESS-mlijfzbu): Executing...
2021-07-23 21:25:21,122 - INFO  - [docker-java-stream-2004020930:DockerUtils$4@347] - DOCKER.exec(PulsarDebeziumSourcesTest-abzgk-pulsar-broker-0:tail -f /var/log/pulsar/broker.log): STDOUT: 21:25:21.000 [pulsar-web-40-6] INFO  org.eclipse.jetty.server.RequestLog - 172.21.0.9 - - [23/Jul/2021:21:25:20 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats?getPreciseBacklog=false&subscriptionBacklogSize=false HTTP/1.1" 200 2476 "-" "Pulsar-Java-v2.9.0-SNAPSHOT" 8
21:25:21.034 [pulsar-web-40-1] INFO  org.eclipse.jetty.server.RequestLog - 172.21.0.9 - - [23/Jul/2021:21:25:21 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats?getPreciseBacklog=false&subscriptionBacklogSize=false HTTP/1.1" 200 2476 "-" "Pulsar-Java-v2.9.0-SNAPSHOT" 16
./build/retry.sh: line 26: 31272 Killed                  "$@"
The command has failed after 3 attempts.

dlg99 · 2021-07-23T22:01:59Z

/pulsarbot run-failure-checks

- check for flush lsn updates

dlg99 · 2021-07-30T21:26:56Z

/pulsarbot run-failure-checks

dlg99 · 2021-07-30T23:42:35Z

Excluded tests for sinks; something times out there, "Pulsar-IO Sinks and Sources" is limited to sources now (but passes)

dlg99 · 2021-07-31T00:34:30Z

/pulsarbot run-failure-checks

Fixes apache#11099 ### Motivation Integration test for debezium connector to tun on CI ### Modifications Debezium integration test for postgres: added check for flush lsn updates Added pulsar-io tests to the CI

Fixes #8502 ### Motivation Upgrade Debezium to a newer version ### Modifications Upgraded Deebzium to v.1.5.4 (latest built with Java 8, v.1.6.x built with Java 11) Upgraded kafka-client to 2.7 (version debezium tested with) Scala-lib to 2.13.6 (for kafka-client) Dealt with API changes, tests etc. PR is on top of #11154 to have Debezium integration tests on CI.

Fixes apache#11099 ### Motivation Integration test for debezium connector to tun on CI ### Modifications Debezium integration test for postgres: added check for flush lsn updates Added pulsar-io tests to the CI

Fixes apache#8502 ### Motivation Upgrade Debezium to a newer version ### Modifications Upgraded Deebzium to v.1.5.4 (latest built with Java 8, v.1.6.x built with Java 11) Upgraded kafka-client to 2.7 (version debezium tested with) Scala-lib to 2.13.6 (for kafka-client) Dealt with API changes, tests etc. PR is on top of apache#11154 to have Debezium integration tests on CI.

dlg99 marked this pull request as draft June 29, 2021 21:13

dlg99 force-pushed the dbz-integration-tests branch 3 times, most recently from 0664a34 to 1cf310b Compare June 30, 2021 18:59

dlg99 marked this pull request as ready for review July 2, 2021 17:51

dlg99 force-pushed the dbz-integration-tests branch from 1cf310b to 7df762f Compare July 2, 2021 18:30

dlg99 mentioned this pull request Jul 2, 2021

[Issue 8502] Upgrade Debezium to a newer version #11204

Merged

1 task

eolivelli approved these changes Jul 3, 2021

View reviewed changes

eolivelli requested review from codelipenghui, merlimat, sijie and lhotari July 6, 2021 07:57

lhotari approved these changes Jul 6, 2021

View reviewed changes

dlg99 force-pushed the dbz-integration-tests branch from 7df762f to 2b16e1c Compare July 7, 2021 16:56

eolivelli mentioned this pull request Jul 13, 2021

[pulsar-io] pass the pulsar service url to debezium source for history database #11251

Merged

1 task

dlg99 force-pushed the dbz-integration-tests branch from 2b16e1c to 5de205b Compare July 15, 2021 02:25

dlg99 force-pushed the dbz-integration-tests branch from 5de205b to e60b186 Compare July 15, 2021 16:29

dlg99 force-pushed the dbz-integration-tests branch from e60b186 to 0b24be0 Compare July 23, 2021 20:27

dlg99 added 2 commits July 29, 2021 12:59

Debezium integration tests

d0610a1

- check for flush lsn updates

Add Pulsar-IO integration test to CI

e0f6a4e

dlg99 force-pushed the dbz-integration-tests branch from 0b24be0 to e0f6a4e Compare July 29, 2021 19:59

dlg99 mentioned this pull request Jul 30, 2021

Debezium Oracle Source #11520

Merged

1 task

exclude tests for sinks

b3c6a2f

sijie assigned dlg99 Aug 4, 2021

sijie added this to the 2.9.0 milestone Aug 4, 2021

sijie approved these changes Aug 4, 2021

View reviewed changes

sijie added area/connector area/test labels Aug 4, 2021

sijie merged commit 5d136a1 into apache:master Aug 4, 2021

dlg99 deleted the dbz-integration-tests branch October 14, 2021 23:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing Debezium integration tests #11154

Fixing Debezium integration tests #11154

dlg99 commented Jun 29, 2021 •

edited

Loading

eolivelli left a comment

lhotari left a comment

dlg99 commented Jul 7, 2021

dlg99 commented Jul 7, 2021

dlg99 commented Jul 7, 2021

eolivelli commented Jul 15, 2021

dlg99 commented Jul 15, 2021

dlg99 commented Jul 16, 2021

nlu90 commented Jul 19, 2021

dlg99 commented Jul 19, 2021

eolivelli commented Jul 20, 2021

eolivelli commented Jul 20, 2021

dlg99 commented Jul 20, 2021

sijie commented Jul 23, 2021

nlu90 commented Jul 23, 2021 •

edited

Loading

dlg99 commented Jul 23, 2021

dlg99 commented Jul 23, 2021

dlg99 commented Jul 30, 2021

dlg99 commented Jul 30, 2021

dlg99 commented Jul 31, 2021

Fixing Debezium integration tests #11154

Fixing Debezium integration tests #11154

Conversation

dlg99 commented Jun 29, 2021 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

eolivelli left a comment

Choose a reason for hiding this comment

lhotari left a comment

Choose a reason for hiding this comment

dlg99 commented Jul 7, 2021

dlg99 commented Jul 7, 2021

dlg99 commented Jul 7, 2021

eolivelli commented Jul 15, 2021

dlg99 commented Jul 15, 2021

dlg99 commented Jul 16, 2021

nlu90 commented Jul 19, 2021

dlg99 commented Jul 19, 2021

eolivelli commented Jul 20, 2021

eolivelli commented Jul 20, 2021

dlg99 commented Jul 20, 2021

sijie commented Jul 23, 2021

nlu90 commented Jul 23, 2021 • edited Loading

dlg99 commented Jul 23, 2021

dlg99 commented Jul 23, 2021

dlg99 commented Jul 30, 2021

dlg99 commented Jul 30, 2021

dlg99 commented Jul 31, 2021

dlg99 commented Jun 29, 2021 •

edited

Loading

nlu90 commented Jul 23, 2021 •

edited

Loading