does OSV works on a local db or using an API?

[tomerse-sg]

Hi,

Does OSV-Scanner works with a local db or with an API?
In case of an API, what are the limitations? how does it handles large assets?
Is it planned to use a local-DB and not an API?

[another-rex]

Does OSV-Scanner works with a local db or with an API?

OSV-Scanner uses the osv.dev API, but we have an experimental local database option: https://google.github.io/osv-scanner/experimental/#local-database-option

The local database option has a few limitations like commit level scanning is not supported, and is potentially slower as all the range matching logic is done locally, and has to download the entire database of an ecosystem before it can start scanning (around 10's of MBs in size, depending on the ecosystem).

In case of an API, what are the limitations? how does it handles large assets?

There is no rate limiting currently on the API, and the speed depends on the query. You'll need to give more details on "large assets" before I can give a more specific answer.

Is it planned to use a local-DB and not an API?

No, the local-db option is an alternative limited option for specific use cases, and there's no plans to make it the default option.

[tomerse-sg]

Hi @another-rex ! thanks for the answer. If I want to find additional data about a CVE \ GHSA \ etc. for enrichment like the API, can I do so using the local DB? using API can cause other issues like API calls limitations (number of calls per minutes)

[another-rex]

Yes, the localdb option uses the OSV.dev data dumps directly, which should container the full published advisory. See the documentation linked in the previous post for more information.

The only difference in the data dump and what the API has access to should be an enumerated list of affected commits.

However, the API should have no rate limits currently, if you do run into being rate limited, please make an issue reporting it, as it is probably a bug!

[tomerse-sg]

great @another-rex ! do you know if it contains all of the CVEs \ GHSA in NVD\Github?
I see in the sources for example red-hat is not covered, however I want general info on the CVE and not in a specific OS context

[another-rex]

It contains all GHSAs (specifically everything that is published to their advisory-database repository + all malicious packages advisories, which should cover all GHSAs, though we have found odd cases where a GHSA isn't published correctly, but that's generally addressed by the Github team pretty quickly).

We do not contain all CVE's, only CVE's related to open source C/C++ projects (@andrewpollock might be able to give more information here). For OS packages, we generally only publish OS specific ones, not general CVEs, as we found general CVEs often contain false positives since OS distros commonly backport fixes to older versions.

[andrewpollock]

Hi @tomerse-sg

https://osv.dev/blog/posts/introducing-broad-c-c++-support/ discusses the CVEs from the NVD in a little more detail. In a nutshell, to be successfully converted, the CVE needs to:

• involve CPEs that a Git repository is known for
  • have discernible fixed versions
  • have derivable commits from the fixed versions

or

• have supported Git URLs as part of the references

If there are specific CVEs of interest to you that are not found to be in OSV, feel free to give me a list and I can take a look. If you have too many, I'll teach you how to determine it for yourself :-)

[tomerse-sg]

thanks for the answer!
does the way osv scarps the information to the db is also an open source that can be reviewed?

thanks!

[another-rex]

Yes, all the source code for ingesting and serving OSV data is located in the OSV.dev repository: https://github.com/google/osv.dev

Specifically the part for ingesting CVEs is located here: https://github.com/google/osv.dev/tree/master/vulnfeeds/cves