Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VAULT-22481] Add audit filtering feature #24558

Merged
merged 17 commits into from
Dec 18, 2023
Merged

Conversation

kubawi
Copy link
Contributor

@kubawi kubawi commented Dec 15, 2023

This PR adds a filtering feature to Vault's audit, which includes:

  • A new filter option in the API and CLI calls to enable an audit device, which accepts go-bexpr expressions.
  • A new type of node for audit pipelines - an entry filter, which decides whether an audit entry should be passed to the further nodes for formatting and writing or it should be filtered out.
  • A bunch of new tests.

For more details see RFC number VLT-291.

Peter Wilson and others added 7 commits December 11, 2023 13:47
* Initial commit on adding filter nodes for audit

* tests for audit filter

* test: longer filter - more conditions

* copywrite headers

* Check interface for the right type
* Support filter nodes in backend factories and add some tests

* More tests and cleanup

* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)

* invert control of the pipelines/nodes to the audit broker vs. within each backend

* update noop audit test code to implement the pipeliner interface

* noop mount path has trailing slash

* attempting to make NoopAudit more friendly

* NoopAudit uses known salt

* Refactor audit.ProcessManual to support filter nodes

* HasFiltering

* rename the pipeliner

* use exported AuditEvent in Filter

* Add tests for registering and deregistering backends on the audit broker

* Add missing licence header to one file, fix a typo in two tests

---------

Co-authored-by: Peter Wilson <[email protected]>
@kubawi kubawi added this to the 1.16.0-rc1 milestone Dec 15, 2023
@kubawi kubawi self-assigned this Dec 15, 2023
@github-actions github-actions bot added the hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed label Dec 15, 2023
Copy link

github-actions bot commented Dec 15, 2023

CI Results:
All Go tests succeeded! ✅

@kubawi kubawi marked this pull request as ready for review December 15, 2023 16:30
@kubawi kubawi requested a review from a team December 15, 2023 16:41
Copy link

Build Results:
All builds succeeded! ✅

Copy link
Contributor

@miagilepner miagilepner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a first pass of comments

audit/entry_filter_test.go Show resolved Hide resolved
// 1. formatter (temporary)
// 2. sink
// 1. filter (optional if configured)
// 2. formatter (temporary)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused by this comment versus the comment on the function. What does it look like if the last node is a filter node? Is the formatter then skipped?

Copy link

@peteski22 peteski22 Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've got a question about whether or not we should allow test messages (which is what would cause ProcessManual to be requested if a filter node is part of the pipeline). So this code may change in a future PR based on the outcomes of that.

To answer the question though 😄 if the filter node filters the message that ends the pipeline, so the formatter and sink nodes never get called.

internal/observability/event/sink_socket.go Show resolved Hide resolved
internal/observability/event/sink_socket_test.go Outdated Show resolved Hide resolved
sdk/logical/audit.go Outdated Show resolved Hide resolved
sdk/logical/audit_test.go Outdated Show resolved Hide resolved
vault/audit_broker_test.go Outdated Show resolved Hide resolved
internal/observability/event/options.go Show resolved Hide resolved
vault/audit_broker_test.go Show resolved Hide resolved
internal/observability/event/pipeline_reader.go Outdated Show resolved Hide resolved
@@ -29,9 +30,19 @@ type SocketSink struct {

// NewSocketSink should be used to create a new SocketSink.
// Accepted options: WithMaxDuration and WithSocketType.
func NewSocketSink(format string, address string, opt ...Option) (*SocketSink, error) {
func NewSocketSink(address string, format string, opt ...Option) (*SocketSink, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since both address and format are of type string, you can do address, format string But not important to get this merged.

internal/observability/event/sink_socket_test.go Outdated Show resolved Hide resolved
vault/audit_broker_test.go Outdated Show resolved Hide resolved
@peteski22 peteski22 enabled auto-merge (squash) December 18, 2023 17:47
@peteski22 peteski22 merged commit 17ffe62 into main Dec 18, 2023
99 checks passed
@peteski22 peteski22 deleted the VAULT-18986/audit-filtering branch December 18, 2023 18:01
Monkeychip pushed a commit that referenced this pull request Jan 7, 2024
* VAULT-22481: Audit filter node (#24465)

* Initial commit on adding filter nodes for audit

* tests for audit filter

* test: longer filter - more conditions

* copywrite headers

* Check interface for the right type

* Add audit filtering feature (#24554)

* Support filter nodes in backend factories and add some tests

* More tests and cleanup

* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)

* invert control of the pipelines/nodes to the audit broker vs. within each backend

* update noop audit test code to implement the pipeliner interface

* noop mount path has trailing slash

* attempting to make NoopAudit more friendly

* NoopAudit uses known salt

* Refactor audit.ProcessManual to support filter nodes

* HasFiltering

* rename the pipeliner

* use exported AuditEvent in Filter

* Add tests for registering and deregistering backends on the audit broker

* Add missing licence header to one file, fix a typo in two tests

---------

Co-authored-by: Peter Wilson <[email protected]>

* Add changelog file

* update bexpr datum to use a strong type

* go docs updates

* test path

* PR review comments

* handle scenarios/outcomes from broker.send

* don't need to re-check the complete sinks

* add extra check to deregister to ensure that re-registering non-filtered device sets sink threshold

* Ensure that the multierror is appended before attempting to return it

---------

Co-authored-by: Peter Wilson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core/audit hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants