[filebeat][gcs] Added support for more mime types, offset tracking via cursor, automatic splitting at root level #34155

ShourieG · 2023-01-02T10:58:11Z

Type of change

Enhancement

What does this PR do?

This PR adds support for more mime types like ndjson, json.gz & json gzipped formats. This also adds support for the following: -

Parsing multiline json files
Offset tracking via cursor state
Automatic splitting at root level object, if this object is an array.

Why is it important?

This greatly improves the functionality of the input and removes potential bugs that could have occurred when used at scale.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Documentation.
Config format.
Code errors.
Design choices.

Related issues

Closes Add support for more MIME types & multiline JSON to GCS input #33982

…level split func

mergify · 2023-01-02T10:58:45Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-01-02T11:02:18Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-01-02T11:04:45Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-23T05:47:19.352+0000
Duration: 76 min 13 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2593
Skipped	168
Total	2761

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6

Initial review

x-pack/filebeat/docs/inputs/input-gcs.asciidoc

x-pack/filebeat/input/gcs/input_test.go

x-pack/filebeat/input/gcs/job.go

x-pack/filebeat/input/gcs/state.go

ShourieG · 2023-01-17T13:34:17Z

@efd6 have updated most of the suggestions except two, for which have left comments.

ShourieG · 2023-01-20T02:21:15Z

@efd6 updated the pr

x-pack/filebeat/input/gcs/job.go

ShourieG · 2023-01-23T05:59:18Z

@efd6 updated the PR with the latest changes

efd6 · 2023-01-23T06:42:19Z

This is an enhancement, so I'm not sure that it should be backported unless there is a specific reason to do so.

ShourieG · 2023-01-23T07:33:54Z

This is an enhancement, so I'm not sure that it should be backported unless there is a specific reason to do so.

@efd6 so there are few customers who are eager to try the gcs integration/input and without the backport they would ideally have to wait till 8.7 to properly use it without issues. Since this is an early beta, I feel we can backport and with the release of 8.6.1 users will get a much more stable input with some essential features that were missing initially.

…a cursor, automatic splitting at root level (#34155) * initial commit -m * added support for more mime type, off set tracking via cursor & root level split func * updated asciidoc * updated NOTICE.txt * updated PR accroding to suggetions * optimised code blocks as per pr suggetions * addressed linting issues * updated with PR suggetions (cherry picked from commit 69ebd98)

…a cursor, automatic splitting at root level (#34155) (#34338) * initial commit -m * added support for more mime type, off set tracking via cursor & root level split func * updated asciidoc * updated NOTICE.txt * updated PR accroding to suggetions * optimised code blocks as per pr suggetions * addressed linting issues * updated with PR suggetions (cherry picked from commit 69ebd98) Co-authored-by: ShourieG <[email protected]>

…a cursor, automatic splitting at root level (#34155) * initial commit -m * added support for more mime type, off set tracking via cursor & root level split func * updated asciidoc * updated NOTICE.txt * updated PR accroding to suggetions * optimised code blocks as per pr suggetions * addressed linting issues * updated with PR suggetions

ShourieG added 3 commits December 31, 2022 00:47

initial commit -m

dcea074

added support for more mime type, off set tracking via cursor & root …

ba2b727

…level split func

Merge remote-tracking branch 'upstream/main' into gcs/add_mime_types

4ab7c03

ShourieG requested a review from a team as a code owner January 2, 2023 10:58

ShourieG requested review from cmacknz and fearful-symmetry and removed request for a team January 2, 2023 10:58

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 2, 2023

updated asciidoc

9743108

mergify bot assigned ShourieG Jan 2, 2023

ShourieG requested a review from a team January 2, 2023 11:02

ShourieG added the Team:Security-External Integrations label Jan 2, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 2, 2023

ShourieG added needs_team Indicates that the issue/PR needs a Team:* label 8.7-candidate labels Jan 2, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 2, 2023

ShourieG added needs_team Indicates that the issue/PR needs a Team:* label backport-v8.6.0 Automated backport with mergify labels Jan 2, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 2, 2023

updated NOTICE.txt

986e878

ShourieG added the enhancement label Jan 2, 2023

ShourieG requested review from efd6 and andrewkroh January 2, 2023 12:28

cmacknz removed request for cmacknz and fearful-symmetry January 4, 2023 19:03

Merge remote-tracking branch 'upstream/main' into gcs/add_mime_types

14e9b4d

ShourieG mentioned this pull request Jan 9, 2023

[GCS] Adding new integration for Custom GCS Input elastic/integrations#4692

Merged

4 tasks

ShourieG added 2 commits January 9, 2023 15:24

Merge remote-tracking branch 'upstream/main' into gcs/add_mime_types

9f1792f

Merge remote-tracking branch 'upstream/main' into gcs/add_mime_types

e9842f8

efd6 reviewed Jan 16, 2023

View reviewed changes

ShourieG added 2 commits January 17, 2023 18:58

updated PR accroding to suggetions

9454617

Merge remote-tracking branch 'upstream/main' into gcs/add_mime_types

44698ff

ShourieG added 3 commits January 19, 2023 11:09

Merge remote-tracking branch 'upstream/main' into gcs/add_mime_types

dfba509

optimised code blocks as per pr suggetions

4a046e5

addressed linting issues

706f087

efd6 reviewed Jan 22, 2023

View reviewed changes

ShourieG added 2 commits January 23, 2023 11:15

updated with PR suggetions

0e4b9d5

Merge remote-tracking branch 'upstream/main' into gcs/add_mime_types

c1c83c3

efd6 approved these changes Jan 23, 2023

View reviewed changes

ShourieG merged commit 69ebd98 into elastic:main Jan 23, 2023

mergify bot mentioned this pull request Jan 23, 2023

[8.6](backport #34155) [filebeat][gcs] Added support for more mime types, offset tracking via cursor, automatic splitting at root level #34338

Merged

ShourieG deleted the gcs/add_mime_types branch January 23, 2023 07:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[filebeat][gcs] Added support for more mime types, offset tracking via cursor, automatic splitting at root level #34155

[filebeat][gcs] Added support for more mime types, offset tracking via cursor, automatic splitting at root level #34155

ShourieG commented Jan 2, 2023

mergify bot commented Jan 2, 2023

elasticmachine commented Jan 2, 2023

elasticmachine commented Jan 2, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

efd6 left a comment

ShourieG commented Jan 17, 2023

ShourieG commented Jan 20, 2023

ShourieG commented Jan 23, 2023

efd6 commented Jan 23, 2023 •

edited

Loading

ShourieG commented Jan 23, 2023

[filebeat][gcs] Added support for more mime types, offset tracking via cursor, automatic splitting at root level #34155

[filebeat][gcs] Added support for more mime types, offset tracking via cursor, automatic splitting at root level #34155

Conversation

ShourieG commented Jan 2, 2023

Type of change

What does this PR do?

Why is it important?

Checklist

Author's Checklist

Related issues

mergify bot commented Jan 2, 2023

elasticmachine commented Jan 2, 2023

elasticmachine commented Jan 2, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

efd6 left a comment

Choose a reason for hiding this comment

ShourieG commented Jan 17, 2023

ShourieG commented Jan 20, 2023

ShourieG commented Jan 23, 2023

efd6 commented Jan 23, 2023 • edited Loading

ShourieG commented Jan 23, 2023

elasticmachine commented Jan 2, 2023 •

edited by jenkins-beats-ci bot

Loading

efd6 commented Jan 23, 2023 •

edited

Loading