[CdapIO] Add complete examples for CdapIO #23501

Amar3tto · 2022-10-05T14:02:55Z

Added examples for the following CDAP plugins:

Hubspot
- Batch Source
- Batch Sink
- Streaming Source
Salesforce
- Batch Source
- Batch Sink
- Streaming Source
ServiceNow
- Batch Source
Zendesk
- Batch Source

Removed "examples" dependency from "hadoop-format-io", because it caused cyclic dependencies issue.
SparkReceiverIO and CdapIO changes:

Added pullFrequencySec parameter - delay in seconds between polling for new records updates.
Added startOffset parameter - inclusive start offset from which the reading should be started.
Refactoring (Plugin, MappingUtils, context classes)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Amar3tto · 2022-10-07T04:58:31Z

Run Java_Examples_Dataflow_Java17 PreCommit

Amar3tto · 2022-10-10T05:52:08Z

Run Java PreCommit

elizaveta-lomteva · 2022-10-12T14:30:37Z

Run Java PreCommit

github-actions · 2022-10-12T15:44:25Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @apilloud for label java.
R: @pabloem for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

github-actions · 2022-10-24T13:44:28Z

Reminder, please take a look at this pr: @apilloud @pabloem

Amar3tto · 2022-10-31T07:07:38Z

Run Java PreCommit

elizaveta-lomteva · 2022-10-31T10:56:18Z

Run Java PreCommit

elizaveta-lomteva · 2022-11-01T08:54:41Z

Run Java PreCommit

codecov · 2022-11-04T15:33:54Z

Codecov Report

Merging #23501 (a4ebc84) into master (63362f5) will decrease coverage by 0.09%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #23501      +/-   ##
==========================================
- Coverage   73.31%   73.22%   -0.10%     
==========================================
  Files         714      717       +3     
  Lines       96418    96932     +514     
==========================================
+ Hits        70686    70974     +288     
- Misses      24411    24637     +226     
  Partials     1321     1321

Flag	Coverage Δ
python	`83.00% <ø> (-0.21%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...l/job_management/v1/beam_expansion_api_pb2_grpc.py	`51.85% <0.00%> (-6.05%)`	⬇️
...hon/apache_beam/runners/worker/worker_pool_main.py	`56.32% <0.00%> (-2.94%)`	⬇️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (-2.44%)`	⬇️
.../python/apache_beam/transforms/periodicsequence.py	`97.01% <0.00%> (-1.50%)`	⬇️
...dks/python/apache_beam/options/pipeline_options.py	`93.96% <0.00%> (-0.94%)`	⬇️
sdks/python/apache_beam/transforms/external.py	`78.88% <0.00%> (-0.86%)`	⬇️
sdks/python/apache_beam/runners/direct/executor.py	`96.46% <0.00%> (-0.55%)`	⬇️
...ache_beam/runners/portability/local_job_service.py	`82.00% <0.00%> (-0.50%)`	⬇️
...apache_beam/typehints/native_type_compatibility.py	`85.16% <0.00%> (-0.37%)`	⬇️
sdks/python/apache_beam/typehints/typehints.py	`93.05% <0.00%> (-0.33%)`	⬇️
... and 31 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

elizaveta-lomteva · 2022-11-17T10:32:01Z

@chamikaramj @aromanenko-dev @mosche @johnjcasey
Hi, happy to announce that we've prepared the PR with examples of using CDAP plugins that we support now for review. The list of supported plugins is in the description.
Thanks a lot for your attention to it!

elizaveta-lomteva · 2022-11-23T17:36:46Z

Run Typescript PreCommit

elizaveta-lomteva · 2022-11-23T17:37:05Z

Run RAT PreCommit

elizaveta-lomteva · 2022-11-23T17:37:16Z

Run Python PreCommit

elizaveta-lomteva · 2022-11-23T17:37:30Z

Run Java PreCommit

Amar3tto · 2022-11-29T11:44:22Z

Run Java_Spark3_Versions PreCommit

elizaveta-lomteva · 2022-11-29T20:01:43Z

R: @aromanenko-dev

elizaveta-lomteva · 2022-11-29T20:02:26Z

R: @johnjcasey

github-actions · 2022-11-29T20:02:59Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

github-actions · 2022-11-29T20:03:26Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

johnjcasey

I don't these examples missing will block CDAP being included the next release. It may be worth splitting this PR into one PR for the IO changes, and one PR for the examples

johnjcasey · 2022-11-30T17:58:46Z

...a/cdap/src/main/java/org/apache/beam/examples/complete/cdap/utils/StructuredRecordUtils.java

+public class StructuredRecordUtils {
+
+  /** Converts {@link StructuredRecord} to String json-like format. */
+  public static String structuredRecordToString(StructuredRecord structuredRecord) {


Is there not a way to do this with json serialization?

johnjcasey · 2022-11-30T18:00:30Z

...les/java/cdap/src/main/java/org/apache/beam/examples/complete/cdap/utils/GetOffsetUtils.java

+ * Class for getting a {@link SerializableFunction} that defines how to get record offset for
+ * different CDAP {@link Plugin} classes.
+ */
+public class GetOffsetUtils {


Why are these utility files organized by function, instead of by plugin?

chamikaramj · 2022-11-30T18:05:14Z

+1 for splitting this huge PR in to multiple PRs for ease of review (may be per example).

elizaveta-lomteva · 2022-11-30T19:15:21Z

@johnjcasey
#24436 is the PR with IO changes
Thank you!

aromanenko-dev · 2022-12-01T15:14:40Z

+1 for multiple PRs split by example

Amar3tto · 2022-12-07T08:24:55Z

Finished splitting PRs:

CdapIO and SparkReceiverIO updates: [CdapIO] CdapIO and SparkReceiverIO updates #24436.
HadoopFormatIO remove examples dependency: [CdapIO] Remove 'examples' dependency from HadoopFormatIO #24564.
Salesforce plugins examples: [CdapIO] Complete examples for CDAP Salesforce plugins #24567.
Hubspot plugins examples: [CdapIO] Complete examples for CDAP Hubspot plugins #24568.
ServiceNow plugins examples: [CdapIO] Complete examples for CDAP ServiceNow plugins #24590.
Zendesk plugins examples: [CdapIO] Complete examples for CDAP Zendesk plugins #24589.

github-actions bot added examples io java hadoop labels Oct 5, 2022

Amar3tto changed the title ~~Add complete examples for CdapIO~~ [CdapIO] Add complete examples for CdapIO Oct 6, 2022

Amar3tto marked this pull request as ready for review October 12, 2022 15:34

github-actions bot added the Next Action: Reviewers label Oct 12, 2022

github-actions bot added the slow-review label Oct 24, 2022

elizaveta-lomteva force-pushed the cdap-io-examples branch from e392e6b to d340f0e Compare October 25, 2022 17:33

github-actions bot removed the slow-review label Oct 31, 2022

Amar3tto force-pushed the cdap-io-examples branch from 19b22f1 to 70cff05 Compare November 3, 2022 10:06

github-actions bot added build and removed build labels Nov 4, 2022

github-actions bot added build and removed build labels Nov 5, 2022

replace api key with auth token

61cf3d5

github-actions bot added build and removed build labels Nov 23, 2022

updated README.md

5996d80

github-actions bot added build and removed build labels Nov 23, 2022

deleted unused dependency

a4ebc84

github-actions bot added build and removed build labels Nov 29, 2022

johnjcasey requested changes Nov 30, 2022

View reviewed changes

elizaveta-lomteva mentioned this pull request Dec 5, 2022

[CdapIO] CdapIO and SparkReceiverIO updates #24436

Merged

4 tasks

Amar3tto closed this Dec 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CdapIO] Add complete examples for CdapIO #23501

[CdapIO] Add complete examples for CdapIO #23501

Amar3tto commented Oct 5, 2022 •

edited

Loading

Amar3tto commented Oct 7, 2022

Amar3tto commented Oct 10, 2022

elizaveta-lomteva commented Oct 12, 2022

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 24, 2022

Amar3tto commented Oct 31, 2022

elizaveta-lomteva commented Oct 31, 2022

elizaveta-lomteva commented Nov 1, 2022

codecov bot commented Nov 4, 2022 •

edited

Loading

elizaveta-lomteva commented Nov 17, 2022

elizaveta-lomteva commented Nov 23, 2022

elizaveta-lomteva commented Nov 23, 2022

elizaveta-lomteva commented Nov 23, 2022

elizaveta-lomteva commented Nov 23, 2022

Amar3tto commented Nov 29, 2022

elizaveta-lomteva commented Nov 29, 2022

elizaveta-lomteva commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

johnjcasey left a comment

johnjcasey Nov 30, 2022

johnjcasey Nov 30, 2022

chamikaramj commented Nov 30, 2022

elizaveta-lomteva commented Nov 30, 2022

aromanenko-dev commented Dec 1, 2022

Amar3tto commented Dec 7, 2022 •

edited

Loading

[CdapIO] Add complete examples for CdapIO #23501

[CdapIO] Add complete examples for CdapIO #23501

Conversation

Amar3tto commented Oct 5, 2022 • edited Loading

GitHub Actions Tests Status (on master branch)

Amar3tto commented Oct 7, 2022

Amar3tto commented Oct 10, 2022

elizaveta-lomteva commented Oct 12, 2022

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 24, 2022

Amar3tto commented Oct 31, 2022

elizaveta-lomteva commented Oct 31, 2022

elizaveta-lomteva commented Nov 1, 2022

codecov bot commented Nov 4, 2022 • edited Loading

Codecov Report

elizaveta-lomteva commented Nov 17, 2022

elizaveta-lomteva commented Nov 23, 2022

elizaveta-lomteva commented Nov 23, 2022

elizaveta-lomteva commented Nov 23, 2022

elizaveta-lomteva commented Nov 23, 2022

Amar3tto commented Nov 29, 2022

elizaveta-lomteva commented Nov 29, 2022

elizaveta-lomteva commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

johnjcasey left a comment

Choose a reason for hiding this comment

johnjcasey Nov 30, 2022

Choose a reason for hiding this comment

johnjcasey Nov 30, 2022

Choose a reason for hiding this comment

chamikaramj commented Nov 30, 2022

elizaveta-lomteva commented Nov 30, 2022

aromanenko-dev commented Dec 1, 2022

Amar3tto commented Dec 7, 2022 • edited Loading

Amar3tto commented Oct 5, 2022 •

edited

Loading

codecov bot commented Nov 4, 2022 •

edited

Loading

Amar3tto commented Dec 7, 2022 •

edited

Loading