Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Kafka topic init task #943

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Introduce Kafka topic init task #943

wants to merge 1 commit into from

Conversation

nscuro
Copy link
Member

@nscuro nscuro commented Oct 3, 2024

Description

Adds a new init task that creates all required topics using their respective default configuration during startup.

This replaces the need for a separate init container (i.e. redpanda-init in our docker-compose.yml: https://github.com/DependencyTrack/hyades/blob/008c3a1b024969a159de085d5ca6b464147faa9a/docker-compose.yml#L251-L273).

The creation of topics is also faster because it can be done in a single batched request, vs. 10s of repetitive CLI invocations.

This init task is disabled by default for now, since it's not meant to be used for production deployments. The idea is to build on this work, and allow users to customize the default topic configurations (e.g. via env vars or config files).

Addressed Issue

Relates to DependencyTrack/hyades#1195

Additional Details

N/A

Checklist

  • I have read and understand the contributing guidelines
  • This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
  • This PR introduces changes to the database model, and I have updated the migration changelog accordingly
  • This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

@nscuro nscuro added enhancement New feature or request dx Issues related to developer experience (DX) labels Oct 3, 2024
nscuro added a commit to DependencyTrack/hyades that referenced this pull request Oct 3, 2024
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <[email protected]>
Copy link

codacy-production bot commented Oct 3, 2024

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
-0.26% (target: -1.00%) 53.85% (target: 70.00%)
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (b0bbd51) 22142 18303 82.66%
Head commit (a8556fb) 22288 (+146) 18365 (+62) 82.40% (-0.26%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#943) 195 105 53.85%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more

nscuro added a commit to DependencyTrack/hyades that referenced this pull request Oct 17, 2024
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <[email protected]>
Adds a new init task that creates all required topics using their respective default configuration during startup.

This replaces the need for a separate init container (i.e. `redpanda-init` in our `docker-compose.yml`: https://github.com/DependencyTrack/hyades/blob/008c3a1b024969a159de085d5ca6b464147faa9a/docker-compose.yml#L251-L273).

The creation of topics is also faster because it can be done in a single batched request, vs. 10s of repetitive CLI invocations.

This init task is disabled by default for now, since it's not meant to be used for production deployments. The idea is to build on this work, and allow users to customize the default topic configurations (e.g. via env vars or config files).

Relates to DependencyTrack/hyades#1195

Signed-off-by: nscuro <[email protected]>
@nscuro nscuro force-pushed the topic-initializer branch from d65dde7 to a8556fb Compare October 17, 2024 17:35
nscuro added a commit to DependencyTrack/hyades that referenced this pull request Oct 19, 2024
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <[email protected]>
nscuro added a commit to DependencyTrack/hyades that referenced this pull request Oct 19, 2024
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dx Issues related to developer experience (DX) enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant