[Filebeat] Add parse_aws_vpc_flow_log processor

[andrewkroh]

What does this PR do?

This is a processor for parsing AWS VPC flow logs. It requires a user specified log format. It can populate the original flow log fields, ECS fields, or both.

Usage:

processors:
  - parse_aws_vpc_flow_log: 
      format: version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
  - community_id: ~

Benchmark:

goos: darwin
goarch: arm64
pkg: github.com/elastic/beats/v7/x-pack/filebeat/processors/aws_vpcflow
BenchmarkProcessorRun/original-mode-v5-message-10                2810948              2138 ns/op            2836 B/op         31 allocs/op
BenchmarkProcessorRun/ecs-mode-v5-message-10                     1914754              3107 ns/op            1908 B/op         41 allocs/op
BenchmarkProcessorRun/ecs_and_original-mode-v5-message-10        1693279              3538 ns/op            3076 B/op         41 allocs/op

Why is it important?

The normal volume of flow logs makes processing them a hot path. So provide a Beat processor to try to make processing as efficient as it can be and make it possible to use the Beat host's CPU to do processing.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Authors Notes

• Need to add static fields like event.kind, event.type, cloud.provider. etc. This can be added by other processors.
• Need to compare output with Fleet package.
  • NOTE: ecs mode in the processor is more aggressive in reducing duplication. The Fleet integration might want to use ecs_and_original then drop the fields needed to maintain its existing behavior.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-11-16T16:09:24.065+0000

• Duration: 104 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	24043
Skipped	1951
Total	25994

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[leehinman]

Nice.

One thing I noticed, in existing implementations you can have a mix of of vpc log formats, as long as they have a different number of fields. Could we change this so it can handle multiple formats? The use case I see is someone modifies their config for recording vpcflow logs, which would result in different formats being in the same S3 bucket.

[andrewkroh]

One thing I noticed, in existing implementations you can have a mix of of vpc log formats, as long as they have a different number of fields. Could we change this so it can handle multiple formats? The use case I see is someone modifies their config for recording vpcflow logs, which would result in different formats being in the same S3 bucket.

I think this was an accidental feature. IMO we wanted to give the users the flexibility of an arbitrary format, but didn't have a way to deliver it due to limitations in ingest pipelines. So instead we loaded the pipeline with a few prescribed formats to get close to the feature.

I can implement this with the requirement that each format have a unique field count. The execution time cost increases by about 4.5% 1.5% when running a config with a single pattern. This would allow the Fleet integration to adopt the feature without a breaking change.

Comparison between 3a6e4cd .. 70bde35

name                                              old time/op    new time/op    delta
ProcessorRun/original-mode-v5-message-10            2.21µs ± 0%    2.14µs ± 0%   ~     (p=1.000 n=1+1)
ProcessorRun/ecs-mode-v5-message-10                 2.80µs ± 0%    2.82µs ± 0%   ~     (p=1.000 n=1+1)
ProcessorRun/ecs_and_original-mode-v5-message-10    3.21µs ± 0%    3.23µs ± 0%   ~     (p=1.000 n=1+1)

[andrewkroh]

@Mergifyio backport 8.6

[mergify]

backport 8.6

✅ Backports have been created

• #33705 [8.6](backport #33656) [Filebeat] Add parse_aws_vpc_flow_log processor has been created for branch 8.6