Mend SCA Parser update

[testaccount90009]

EDIT - I will publish a different PR with multiple Parser files for one Parser similar to Sonarqube structure. Doing them all in one file is not neat and driving me crazy :). Code findings in API 3.0 / Platform for Mend, SAST respectively.

Minor Platform SCA and Legacy SCA edits.

Part one of this PR is to update 'Locations' and 'File Paths' to try and manage a cleaner way to extract the Path for legacy Mend SCA format vulnerabilities as mentioned here - #11340

Part two of this PR is to remove Locations from Description in the Mend Platform SCA addition I most recently added in 2.40.0.

EDIT - See Edit statement above, but this 'Part 3' for SAST will be contributed after I refactor the parser to have separate files for each, though still retaining one single Parser option. Doing this all in one file is not ideal.
Part three of this PR is to add Mend Platform SAST / 3.0 (Code Security Findings / Vulnerabilities) formatted Findings.

The idea is to create one parser that can accomplish:
-Original Mend Parser SCA Findings preservation. I do not want to update or edit this here much, since people may already be using this in the wild. Previously I added an edit to the Locations / File Paths, but due to the issue linked above; it may negatively impact teams who have a large list of file paths / locations with their Findings from the Mend Legacy SCA (non-platform) output / legacy 1.0 API formatted output.

-New Platform SCA parser. This was implemented in my most recent PR affecting Mend parser.py, however an oversight of adding the Filepath to the Description has me wanting to refactor the Location out of the Description, as to not affect deduplication since Description and Title are used.

EDIT - See above Edits, this isn't in this PR at this moment.
-New Platform SAST parser. This will be the 'meat' and main portion of this PR. Code SAST Findings in Mend should be able to be identified and structured as a result of changes in this PR.

Checklist

This checklist is for your information.

• Make sure to rebase your PR against the very latest dev.
• Features/Changes should be submitted against the dev.
• Bugfixes should be submitted against the bugfix branch.
• Give a meaningful name to your PR, as it may end up being used in the release notes.
• Your code is flake8 compliant.
• Your code is python 3.11 compliant.
• If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
• Model changes must include the necessary migrations in the dojo/db_migrations folder.
• Add applicable tests to the unit tests.
• Add the proper label to categorize your PR.

Extra information

Please clear everything below when submitting your pull request, it's here purely for your information.

Moderators: Labels currently accepted for PRs:

• Import Scans (for new scanners/importers)
• enhancement
• performance
• feature
• bugfix
• maintenance (a.k.a chores)
• dependencies
• New Migration (when the PR introduces a DB migration)
• settings_changes (when the PR introduces changes or new settings in settings.dist.py)

Contributors: Git Tips

Rebase on dev branch

If the dev branch has changed since you started working on it, please rebase your work after the current dev.

On your working branch mybranch:

git rebase dev mybranch

In case of conflict:

 git mergetool
 git rebase --continue

When everything's fine on your local branch, force push to your myOrigin remote:

git push myOrigin --force-with-lease

To cancel everything:

git rebase --abort

Squashing commits

git rebase -i origin/dev

• Replace pick by fixup on the commits you want squashed out
• Replace pick by reword on the first commit if you want to change the commit message
• Save the file and quit your editor

Force push to your myOrigin remote:

git push myOrigin --force-with-lease

[dryrunsecurity]

DryRun Security Summary

The pull request aims to improve the Defect Dojo application's integration with the Mend SCA tool by enhancing vulnerability data handling, updating unit tests, and ensuring more comprehensive and accurate security risk management.

Expand for full summary

Summary:

The code changes in this pull request are focused on improving the functionality and security of the Defect Dojo application's integration with the Mend (previously known as WhiteSource) Software Composition Analysis (SCA) tool. The changes address several key areas:

1. Unit Test Updates: The changes in the test_mend_parser.py file update the assertions in a specific test case to ensure the expected output of the MendParser class. While this change is not directly related to security, it is important to maintain a robust test suite to catch any potential regressions or issues early in the development process.

2. Handling of Vulnerability Data: The changes in the dojo/tools/mend/parser.py file focus on improving the handling of vulnerability data from the Mend SCA tool. This includes truncating long "locations" fields, populating the "steps_to_reproduce" field with location information, and better handling the "topFix" field to extract mitigation information. These changes help ensure that the vulnerability data imported into Defect Dojo is more comprehensive and accurate, which is crucial for effective security risk management.

Overall, these code changes are focused on improving the security posture of the Defect Dojo application by enhancing the integration with the Mend SCA tool and providing more complete and accurate vulnerability data. The changes do not introduce any obvious security vulnerabilities and are a positive step towards strengthening the application's security capabilities.

Files Changed:

1. unittests/tools/test_mend_parser.py: The changes in this file update the assertion for the finding.file_path attribute in the test_parse_file_with_one_sca_vuln_finding test case. This is a non-security-related change to the unit test suite.

2. dojo/tools/mend/parser.py: The changes in this file focus on improving the handling of vulnerability data from the Mend SCA tool. Specifically, the changes address the following:

   • Truncating long "locations" fields to ensure they fit within the Finding model's character limit.
   • Populating the "steps_to_reproduce" field with the list of locations for the vulnerability.
   • Better handling of the "topFix" field to extract mitigation information.

Code Analysis

We ran 9 analyzers against 2 files and 0 analyzers had findings. 9 analyzers had no findings.

View PR in the DryRun Dashboard.

[testaccount90009]

To be clear, this is only for SCA and fixing Locations + removing the Library Path from the description to avoid being used in deduplication.

The mentioned SAST changes will be in a future PR after I refactor and clean up this so I can break out a single parser into multiple files, instead of attempting to code everything in one file.

I will refactor like this structure -
https://github.com/DefectDojo/django-DefectDojo/tree/dev/dojo/tools/sonarqube

Instead of one single parser.py file attempting to construct each output case.

[testaccount90009]

I am testing these changes in my dev stack right now.

[testaccount90009]

After testing import on Legacy SCA and Platform SCA:

Legacy - These changes fix the varchar 4000 issue for the file paths field for legacy SCA vulns, while adding it into steps_to_reproduce successfully. The file_paths is no longer used for Legacy SCA and can no longer error out on the var char 4000 limit for that field in the DB/model.

Platform - Adding logic for steps_to_reproduce to retrieve the component.path and Transitive dependencies should now include the component -> path as the steps_to_reproduce. If the library and vulnerability is Direct, then Path is not in the Mend SCA Platform output json file, only for Transitive ones for some reason. Maybe they (Mend) will update this at some point, but it's still good to gather the component path the vulnerability is in and whenever they get around to adding Direct dependencies with the component path data, it will be available in the Finding in DefectDojo with this parser change.\

I am pretty sure this is the final Legacy / SCA Platform parser change I am setting out to accomplish, before my next PR which will break this out similar to a structure as defined in the sonarqube tool parser. One parser file to handle the context of multiple schemas is not the cleanest approach, and after talking with my team - we agreed that before I introduce the Platform SAST parser capabilities, it would be best to restructure the Mend parser files to be similar to that of SonarQube and making the logic a bit easier to follow, when broken out.

[testaccount90009]

I tested both the legacy SCA (original) and platform SCA findings and it looks like locations are correctly added to the steps_to_reproduce block now. I welcome any feedback or input on these set of changes.

[testaccount90009]

It's worth noting that in the SCA Platform uploads, some of the json files are missing component -> path and cannot pull the location into steps to reproduce field. This is a bug with Mend that I am bringing up with their support team, as their Platform UI does not show the location either - whereas their Legacy one does... This means it's there, just some kind of bug on their backend for providing that Location data into the Platform UI + json output. Assuming Mend fixes Direct dependencies to have a Location (only works for Transitive, pulling in the Direct dependency via dependencyFile) then the steps to reproduce will eventually start picking up the Locations of Direct dependencies, once Mend identifies a fix.

In a nutshell: if Mend scanned something and identified a library - it has to know where the location is (whether direct match or dep file inclusion). So by design, that should be included and since it's in the Legacy UI + json, it's possible -- so something in their Platform is broken.