Use server-side filtering when retrieving Cloud Foundry logs #33456

jsoriano · 2022-10-25T15:37:01Z

What does this PR do?

Use server-side filtering when retrieving Cloud Foundry logs.

Why is it important?

It reduces CPU usage of Filebeat significatively.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Run Cloud Foundry input with firehose_filter: logs, check that log messages are still collected.

Related issues

Use cases

Screenshots

Logs

botelastic · 2022-10-25T15:37:08Z

This pull request doesn't have a Team:<team> label.

mergify · 2022-10-25T15:37:36Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @jsoriano? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2022-10-25T15:43:14Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-11-08T09:02:23.793+0000
Duration: 74 min 45 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7058
Skipped	335
Total	7393

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2022-10-27T02:28:24Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b cloudfoundry-config-filter upstream/cloudfoundry-config-filter
git merge upstream/main
git push upstream cloudfoundry-config-filter

…-filter

jsoriano · 2022-11-07T14:18:53Z

/package

jsoriano

@MichaelKatsoulis @gizas if both logs modes collect the same logs, I wonder if we should just use the new method with server and client filtering, and don't expose the option to user. Wdyt?

jsoriano · 2022-11-07T14:35:08Z

x-pack/libbeat/common/cloudfoundry/dopplerconsumer.go

+		// TODO: Handle unknown filters.
+		return filterNoFilter, -1


We should decide if we want some kind of error handling here, or if we just consider all the default in this case.

jsoriano · 2022-11-07T14:35:23Z

x-pack/libbeat/common/cloudfoundry/dopplerconsumer.go

+		return filterLogs, -1
+	case FirehoseFilterLogs:
+		// Uses filter-type=logs in requests to the firehose.
+		return filterLogs, consumer.LogMessages


LGTM to use both filters here.

jsoriano · 2022-11-07T14:36:46Z

x-pack/libbeat/common/cloudfoundry/dopplerconsumer.go

@@ -80,23 +81,49 @@ func (c *DopplerConsumer) Run() {
 }

 func (c *DopplerConsumer) logsFirehose() {
-	c.firehose(c.callbacks.Log, consumer.LogMessages)
+	filter := FirehoseFilterAllLogs


If we are sure that FirehoseFilterAllLogs and FirehoseFilterLogs collect the same logs, but FirehoseFilterLogs is much more efficient, wdyt about using FirehoseFilterLogs here and don't expose the option to the user?

Suggested change

filter := FirehoseFilterAllLogs

filter := FirehoseFilterLogs

MichaelKatsoulis · 2022-11-07T17:21:07Z

@MichaelKatsoulis @gizas if both logs modes collect the same logs, I wonder if we should just use the new method with server and client filtering, and don't expose the option to user. Wdyt?

+1 on that. Let's not expose the logs vs all-logs. Let's keep all-logs with server and client side filtering , but expose the other options

…ilter

jsoriano · 2022-11-07T18:53:54Z

Let's not expose the logs vs all-logs. Let's keep all-logs with server and client side filtering , but expose the other options

And given that, shall we be exposing anything at all regarding this?

I have pushed a couple of changes, to remove all-logs vs. logs and to completely remove the setting and simplify the code, let me know what you think.

mergify · 2022-11-08T07:21:36Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b cloudfoundry-config-filter upstream/cloudfoundry-config-filter
git merge upstream/main
git push upstream cloudfoundry-config-filter

…-filter

MichaelKatsoulis · 2022-11-08T09:03:12Z

And given that, shall we be exposing anything at all regarding this?

I agree. Let's keep it simpler. I will test it and then we can merge it

MichaelKatsoulis · 2022-11-08T10:59:07Z

@jsoriano it works as expected. We can merge when ci passes

It reduces CPU usage of Filebeat significatively. Co-authored-by: MichaelKatsoulis <[email protected]> (cherry picked from commit 5cd8b09)

…ud Foundry logs (#33637) * Use server-side filtering when retrieving Cloud Foundry logs (#33456) It reduces CPU usage of Filebeat significatively. Co-authored-by: MichaelKatsoulis <[email protected]> (cherry picked from commit 5cd8b09) * Update CHANGELOG.next.asciidoc Co-authored-by: Jaime Soriano Pastor <[email protected]> Co-authored-by: Michael Katsoulis <[email protected]>

It reduces CPU usage of Filebeat significatively. Co-authored-by: MichaelKatsoulis <[email protected]>

Expose Cloud Foundry filtering into user configuration

971dd5d

jsoriano requested a review from a team October 25, 2022 15:37

jsoriano self-assigned this Oct 25, 2022

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 25, 2022

Fix changelog

20c7ddc

Linting and fix on docs

7fc340d

MichaelKatsoulis approved these changes Oct 26, 2022

View reviewed changes

MichaelKatsoulis added 3 commits November 7, 2022 12:21

Add both server and client side filtering

fae0b0a

Merge remote-tracking branch 'upstream/main' into cloudfoundry-config…

1a1ff0f

…-filter

Edit changelog

989dfde

MichaelKatsoulis marked this pull request as ready for review November 7, 2022 10:26

MichaelKatsoulis requested a review from a team as a code owner November 7, 2022 10:26

MichaelKatsoulis requested review from fearful-symmetry and faec and removed request for a team November 7, 2022 10:26

jsoriano commented Nov 7, 2022

View reviewed changes

jsoriano added 3 commits November 7, 2022 19:41

Merge remote-tracking branch 'origin/main' into cloudfoundry-config-f…

7b6afaf

…ilter

Remove all-logs option

17d262b

Remove option, simplify

3deb848

MichaelKatsoulis added 2 commits November 8, 2022 11:00

Merge remote-tracking branch 'upstream/main' into cloudfoundry-config…

07e002b

…-filter

Merge upstream

f0ca944

jsoriano changed the title ~~Expose Cloud Foundry filtering into user configuration~~ Use server-side filtering when retrieving Cloud Foundry logs Nov 8, 2022

jsoriano merged commit 5cd8b09 into elastic:main Nov 8, 2022

jsoriano deleted the cloudfoundry-config-filter branch November 8, 2022 11:01

MichaelKatsoulis added the backport-7.17 Automated backport to the 7.17 branch with mergify label Nov 10, 2022

mergify bot mentioned this pull request Nov 10, 2022

[7.17](backport #33456) Use server-side filtering when retrieving Cloud Foundry logs #33637

Merged

mergify bot pushed a commit that referenced this pull request Nov 10, 2022

Use server-side filtering when retrieving Cloud Foundry logs (#33456)

421874f

It reduces CPU usage of Filebeat significatively. Co-authored-by: MichaelKatsoulis <[email protected]> (cherry picked from commit 5cd8b09)

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

Use server-side filtering when retrieving Cloud Foundry logs (#33456)

43336e7

It reduces CPU usage of Filebeat significatively. Co-authored-by: MichaelKatsoulis <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use server-side filtering when retrieving Cloud Foundry logs #33456

Use server-side filtering when retrieving Cloud Foundry logs #33456

jsoriano commented Oct 25, 2022 •

edited

Loading

botelastic bot commented Oct 25, 2022

mergify bot commented Oct 25, 2022

elasticmachine commented Oct 25, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

mergify bot commented Oct 27, 2022

jsoriano commented Nov 7, 2022

jsoriano left a comment

jsoriano Nov 7, 2022

jsoriano Nov 7, 2022

jsoriano Nov 7, 2022

MichaelKatsoulis commented Nov 7, 2022

jsoriano commented Nov 7, 2022

mergify bot commented Nov 8, 2022

MichaelKatsoulis commented Nov 8, 2022

MichaelKatsoulis commented Nov 8, 2022

Use server-side filtering when retrieving Cloud Foundry logs #33456

Use server-side filtering when retrieving Cloud Foundry logs #33456

Conversation

jsoriano commented Oct 25, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

botelastic bot commented Oct 25, 2022

mergify bot commented Oct 25, 2022

elasticmachine commented Oct 25, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Oct 27, 2022

jsoriano commented Nov 7, 2022

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Nov 7, 2022

Choose a reason for hiding this comment

jsoriano Nov 7, 2022

Choose a reason for hiding this comment

jsoriano Nov 7, 2022

Choose a reason for hiding this comment

MichaelKatsoulis commented Nov 7, 2022

jsoriano commented Nov 7, 2022

mergify bot commented Nov 8, 2022

MichaelKatsoulis commented Nov 8, 2022

MichaelKatsoulis commented Nov 8, 2022

jsoriano commented Oct 25, 2022 •

edited

Loading

elasticmachine commented Oct 25, 2022 •

edited by jenkins-beats-ci bot

Loading