Fix typing in more places and configure mypy to follow imports

[dekkers]

Changes

This contains a lot of fixes for missing or in some cases wrong typing. There are also a few small bugfixes in cases where there was a missing check for None.

Mypy is configured to follow imports now which means it will be able to catch a lot more issues. Only pydantic imports are not followed because that would generate too many errors.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[underdarknl]

This Pr is going to cause a lot of merge conflicts with Donny's new boefjes / container work I think.

[ammar92]

Looks good to me

[dekkers]

This Pr is going to cause a lot of merge conflicts with Donny's new boefjes / container work I think.

I assume you mean #3859 with that, but I just tried merging this branch into that branch and the only conflicts I got were the ones that already are there with current main.

[stephanie0x00]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make reset.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.
• I checked the logs for errors and/or warnings and made issues where necessary

What works:

• CSV upload with multiple hosts
• Raw file upload (burp file)
• Onboarding + onboarding report
• Enabled all boefjes that do not require an API key
• Yielded objects under Tasks -> Normalizers are present and work.
• Opening subreports for aggregate and normal reports
• Opening main reports (normal and aggregate)
• Scheduling of reports
• Creating of new member

What doesn't work:

See below, two boefjes throw an error.

Bug or feature?:

CWE finding types

Traceback (most recent call last):
  File "/app/boefjes/boefjes/local.py", line 58, in run
    return boefje_resource.module.run(boefje_meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/boefjes/boefjes/plugins/kat_cwe_finding_types/main.py", line 24, in run
    name = weakness_elem.get("Name")
           ^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/boefjes/boefjes/job_handler.py", line 144, in handle
    boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/boefjes/boefjes/local.py", line 60, in run
    raise JobRuntimeError("Boefje failed") from e
boefjes.runtime_interfaces.JobRuntimeError: Boefje failed

DNS Software version

Traceback (most recent call last):
  File "/app/boefjes/boefjes/local.py", line 58, in run
    return boefje_resource.module.run(boefje_meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/boefjes/boefjes/plugins/kat_dns_version/main.py", line 34, in run
    response = method(query, where=ip, timeout=timeout, port=port)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dns/query.py", line 1028, in tcp
    (r, received_time) = receive_tcp(
                         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dns/query.py", line 947, in receive_tcp
    ldata = _net_read(sock, 2, expiration)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dns/query.py", line 858, in _net_read
    raise EOFError
EOFError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/boefjes/boefjes/job_handler.py", line 144, in handle
    boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/boefjes/boefjes/local.py", line 60, in run
    raise JobRuntimeError("Boefje failed") from e
boefjes.runtime_interfaces.JobRuntimeError: Boefje failed

[ammar92]

See below, two boefjes throw an error.

Both errors seem not be related to this PR.

CWE finding types

Seems to be an error related to a CWE ID not being found in our local database. Perhaps the database is out of date and is now fixed by #3943?

DNS Software version

This error seems to be a runtime error that's not caught (either a network error while reading the socket, or abrupt closure of the connection by the remote service)

[underdarknl]

See below, two boefjes throw an error.

Both errors seem not be related to this PR.

CWE finding types

Seems to be an error related to a CWE ID not being found in our local database. Perhaps the database is out of date and is now fixed by #3943?

DNS Software version

This error seems to be a runtime error that's not caught (either a network error while reading the socket, or abrupt closure of the connection by the remote service)

We should probably also catch the exception of the get() on the none or by doing a proper lookup of the expected key. Not being able to hydrate this CWE should be visible by telling the user we don't have information for the given CWE.

[stephanie0x00]

See below, two boefjes throw an error.

Both errors seem not be related to this PR.

CWE finding types

Seems to be an error related to a CWE ID not being found in our local database. Perhaps the database is out of date and is now fixed by #3943?

Sadly not, I just ran into this error in #3943.

[ammar92]

Sadly not, I just ran into this error in #3943.

A raw file could help debugging this

[stephanie0x00]

As discussed with @ammar92 the CWE bug is resolved in #3943. As for the DNS Software version I'll keep an eye out and see if I can trigger this again. This ticket can be merged.

[sonarcloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
88.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud