qualys_vmdr: Improve support for Cloud Detection and Response (CDR) workflow #11999

kcreddy · 2024-12-04T15:17:32Z

Proposed commit message

Add Qualys VMDR's Asset data to Elastic's Cloud Native Vulnerability Management (CNVM) workflow.

Note

To Reviewers:
Please refer to this guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide which was used for proposed changes.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Pipeline Tests:
Run:

cd packages/qualys_vmdr && elastic-package build && elastic-package stack up --version=8.16.0 -d -v  && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=asset_host_detection

Returns success:

--- Test results for package: qualys_vmdr - START ---
╭─────────────┬──────────────────────┬───────────┬──────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM          │ TEST TYPE │ TEST NAME                                                │ RESULT │ TIME ELAPSED │
├─────────────┼──────────────────────┼───────────┼──────────────────────────────────────────────────────────┼────────┼──────────────┤
│ qualys_vmdr │ asset_host_detection │ pipeline  │ (ingest pipeline warnings test-asset-host-detection.log) │ PASS   │ 319.428583ms │
│ qualys_vmdr │ asset_host_detection │ pipeline  │ test-asset-host-detection.log                            │ PASS   │ 306.113459ms │
╰─────────────┴──────────────────────┴───────────┴──────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: qualys_vmdr - END   ---
Done

System Tests:
Run:

cd packages/qualys_vmdr && elastic-package build && elastic-package stack up --version=8.16.0 -d -v  && eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v --data-streams=asset_host_detection

Returns success:

--- Test results for package: qualys_vmdr - START ---
╭─────────────┬──────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE     │ DATA STREAM          │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────────┼──────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ qualys_vmdr │ asset_host_detection │ system    │ ahd-kb    │ PASS   │ 39.780903291s │
╰─────────────┴──────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: qualys_vmdr - END   ---
Done

Related issues

Relates [meta] Qualys VMDR: Enhancement to leverage cloud workflows #11673

Screenshots

Source

Destination

Transform

elasticmachine · 2024-12-10T10:32:24Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elasticmachine · 2024-12-10T11:24:50Z

💚 Build Succeeded

Buildkite Build
Commit: c2f8a80

History

💚 Build #19277 succeeded 268bf22
💚 Build #19274 succeeded 8dcb251
💔 Build #19269 failed ded958c
💔 Build #19222 failed 5479804
💔 Build #19221 failed 05744eb
💔 Build #19219 failed 6b07bf8

cc @kcreddy

elastic-sonarqube · 2024-12-10T11:24:57Z

Quality Gate failed

Failed conditions
60.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

kcreddy · 2024-12-10T11:25:09Z

packages/qualys_vmdr/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

+    - resource.id
+    - event.id
+    - data_stream.namespace


This list of unique_key fields is different from the proposed guide due to the issue described in here and applying follow-up suggestion from here.

efd6 · 2024-12-11T22:05:32Z

packages/qualys_vmdr/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "6.0.0"
+  changes:
+    - description: Leverage Cloud Detection and Response (CDR) workflows in asset_host_detection data-stream.


Suggested change

- description: Leverage Cloud Detection and Response (CDR) workflows in asset_host_detection data-stream.

- description: Use Cloud Detection and Response (CDR) workflows in asset_host_detection data-stream.

?

efd6 · 2024-12-11T22:11:25Z

packages/qualys_vmdr/data_stream/asset_host_detection/elasticsearch/ingest_pipeline/default.yml

@@ -1066,6 +1142,70 @@ processors:
      tag: rename_qualys_vmdr_asset_host_detection_vulnerability_RESULTS_1
      target_field: qualys_vmdr.asset_host_detection.vulnerability.results
      ignore_missing: true
+  # The below 2 gsub processors facilitate `splitOnToken` in subsequent script processor which otherwise cannot split \n or \t.


Suggested change

# The below 2 gsub processors facilitate `splitOnToken` in subsequent script processor which otherwise cannot split \n or \t.

# The next two gsub processors facilitate `splitOnToken` in the subsequent script processor, which otherwise cannot split \n or \t.

Do we know that ; and | do not otherwise appear in the text?

efd6 · 2024-12-11T22:15:31Z

...r/data_stream/asset_host_detection/elasticsearch/ingest_pipeline/pipeline_knowledge_base.yml

+          return;
+        }
+        def level = Long.parseLong(ctx.qualys_vmdr.asset_host_detection.knowledge_base.SEVERITY_LEVEL);
+        if (['Potential Vulnerability', 'Vulnerability', 'Vulnerability or Potential Vulnerability'].contains(vuln_type)) {


Could this array be put in a params array to avoid allocs?

efd6 · 2024-12-11T22:16:35Z

...r/data_stream/asset_host_detection/elasticsearch/ingest_pipeline/pipeline_knowledge_base.yml

+          if (level == 1){
+              ctx.qualys_vmdr.asset_host_detection.knowledge_base.SEVERITY_LEVEL = "Minimal";
+          } else if (level == 2) {
+              ctx.qualys_vmdr.asset_host_detection.knowledge_base.SEVERITY_LEVEL = "Medium";
+          } else if (level == 3) {
+              ctx.qualys_vmdr.asset_host_detection.knowledge_base.SEVERITY_LEVEL = "Serious";
+          } else if (level == 4) {
+              ctx.qualys_vmdr.asset_host_detection.knowledge_base.SEVERITY_LEVEL = "Critical";
+          } else if (level == 5) {
+              ctx.qualys_vmdr.asset_host_detection.knowledge_base.SEVERITY_LEVEL = "Urgent";
+          }


These are dense with a numeric index, so the assignment could be done from a params array after bounds checking. Same below.

efd6 · 2024-12-11T22:18:24Z

packages/qualys_vmdr/data_stream/asset_host_detection/elasticsearch/ingest_pipeline/default.yml

Add a

- append: field: tags value: preserve_original_event allow_duplicates: false

to the final on_failure.

efd6 · 2024-12-11T22:24:42Z

packages/qualys_vmdr/data_stream/asset_host_detection/agent/stream/input.yml.hbs

+                "Authorization": ["Basic "+(state.user+":"+state.password).base64()],
+            }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+            resp.Body.as(xml, try(bytes(xml).decode_xml('qualys_api_2_0_ahd'), "decode_xml_error_ahd").as(ahd_body, 


Since we are now on v8.16.0 this can be

Suggested change

resp.Body.as(xml, try(bytes(xml).decode_xml('qualys_api_2_0_ahd'), "decode_xml_error_ahd").as(ahd_body,

resp.Body.as(xml, try(xml.decode_xml('qualys_api_2_0_ahd'), "decode_xml_error_ahd").as(ahd_body,

efd6 · 2024-12-11T22:25:03Z

packages/qualys_vmdr/data_stream/asset_host_detection/agent/stream/input.yml.hbs

-            (body.?doc.HOST_LIST_VM_DETECTION_OUTPUT.RESPONSE.HOST_LIST.HOST.hasValue() 
+          }
+        }).do_request().as(resp, resp.StatusCode == 200 ?
+          resp.Body.as(xml, try(bytes(xml).decode_xml('qualys_api_2_0_kb'), "decode_xml_error_kb").as(kb_body, 


Suggested change

resp.Body.as(xml, try(bytes(xml).decode_xml('qualys_api_2_0_kb'), "decode_xml_error_kb").as(kb_body,

resp.Body.as(xml, try(xml.decode_xml('qualys_api_2_0_kb'), "decode_xml_error_kb").as(kb_body,

kcreddy added 2 commits December 4, 2024 16:34

Add CEL input and system test

76ffb86

Add knowledge_base pipeline and update sys tests

5836c89

kcreddy mentioned this pull request Dec 4, 2024

[meta] Qualys VMDR: Enhancement to leverage cloud workflows #11673

Open

10 tasks

andrewkroh added the Integration:qualys_vmdr Qualys VMDR label Dec 4, 2024

kcreddy added 8 commits December 9, 2024 01:29

Update mappings and pipeline tests

92c9f84

sanitize and udpate pipeline tests

fd4d9a4

Fix pipeline tests

67b0a20

Add transform and ecs mappings

c984e90

Add package.* and vulnerability.package.*

6b07bf8

Merge remote-tracking branch 'upstream/main' into qualys-cdr

05744eb

Update system tests

5479804

FIx system tests and update transform

ded958c

kcreddy added the enhancement New feature or request label Dec 10, 2024

kcreddy self-assigned this Dec 10, 2024

kcreddy added the Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] label Dec 10, 2024

kcreddy added 3 commits December 10, 2024 13:29

Fix pipeline tests

8dcb251

Add different approach to worklist

c76a843

Update transform unique key and add query

268bf22

kcreddy marked this pull request as ready for review December 10, 2024 10:32

kcreddy requested a review from a team as a code owner December 10, 2024 10:32

update pr number

c2f8a80

kcreddy requested a review from a team December 10, 2024 11:09

kcreddy commented Dec 10, 2024

View reviewed changes

efd6 reviewed Dec 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

qualys_vmdr: Improve support for Cloud Detection and Response (CDR) workflow #11999

qualys_vmdr: Improve support for Cloud Detection and Response (CDR) workflow #11999

kcreddy commented Dec 4, 2024 •

edited

Loading

elasticmachine commented Dec 10, 2024

elasticmachine commented Dec 10, 2024

elastic-sonarqube bot commented Dec 10, 2024

kcreddy Dec 10, 2024 •

edited

Loading

efd6 Dec 11, 2024

efd6 Dec 11, 2024

efd6 Dec 11, 2024

efd6 Dec 11, 2024

efd6 Dec 11, 2024

efd6 Dec 11, 2024

efd6 Dec 11, 2024

	- description: Leverage Cloud Detection and Response (CDR) workflows in asset_host_detection data-stream.
	- description: Use Cloud Detection and Response (CDR) workflows in asset_host_detection data-stream.

	# The below 2 gsub processors facilitate `splitOnToken` in subsequent script processor which otherwise cannot split \n or \t.
	# The next two gsub processors facilitate `splitOnToken` in the subsequent script processor, which otherwise cannot split \n or \t.

	resp.Body.as(xml, try(bytes(xml).decode_xml('qualys_api_2_0_ahd'), "decode_xml_error_ahd").as(ahd_body,
	resp.Body.as(xml, try(xml.decode_xml('qualys_api_2_0_ahd'), "decode_xml_error_ahd").as(ahd_body,

qualys_vmdr: Improve support for Cloud Detection and Response (CDR) workflow #11999

Are you sure you want to change the base?

qualys_vmdr: Improve support for Cloud Detection and Response (CDR) workflow #11999

Conversation

kcreddy commented Dec 4, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Dec 10, 2024

elasticmachine commented Dec 10, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Dec 10, 2024

Quality Gate failed

kcreddy Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

efd6 Dec 11, 2024

Choose a reason for hiding this comment

efd6 Dec 11, 2024

Choose a reason for hiding this comment

efd6 Dec 11, 2024

Choose a reason for hiding this comment

efd6 Dec 11, 2024

Choose a reason for hiding this comment

efd6 Dec 11, 2024

Choose a reason for hiding this comment

efd6 Dec 11, 2024

Choose a reason for hiding this comment

efd6 Dec 11, 2024

Choose a reason for hiding this comment

kcreddy commented Dec 4, 2024 •

edited

Loading

kcreddy Dec 10, 2024 •

edited

Loading