sbomqs score directly from git URLs

[viveksahu26]

closes: #266
This PR add support for sbomqs score command to score directly from git URLs. Till now it funtionality limited to local files. For example:
$ sbomqs score -b <sbom.spdx.json file>

New feature supports:

$ sbomqs score -b  https://github.com/interlynk-io/sbomqs/blob/main/samples/
or 
$ sbomqs score -b https://github.com/interlynk-io/sbomqs/blob/main/samples/sbomqs-spdx-syft.json
or 
$ sbomqs score -b  https://github.com/spdx/ntia-conformance-checker/blob/main/tests/data/ -b 
or 
$ sbomqs score -b  https://github.com/spdx/ntia-conformance-checker/blob/main/tests/data/SPDXSBOMExampleTests/ -b

Example:

$ go run main.go score https://github.com/interlynk-io/sbomqs/blob/main/samples/ -b
Enumerating objects: 561, done.
Counting objects: 100% (561/561), done.
Compressing objects: 100% (354/354), done.
Total 561 (delta 363), reused 330 (delta 190), pack-reused 0
7.7     spdx    2.3     json    samples/photon.spdx.json
6.5     cdx     1.4     json    samples/sbomqs-cdx-cgomod.json
6.5     spdx    2.3     json    samples/sbomqs-spdx-sbtool.json
6.4     spdx    2.3     json    samples/sbomqs-spdx-syft.json

[riteshnoronha]

@viveksahu26 not sure cloning the entire repo just to get the files is a good idea. Is there a way just to pull files that are sboms. We can fix what extensions we look at.

[viveksahu26]

So, yeah @riteshnoronha . It is possible to implement in a different way i.e. apart form cloning whole repo:

• Basically fetching URL directly. Which gives the information related to files and it's content. And then writing it's content to a file in a memory instead of local storage.

[riteshnoronha]

@viveksahu26 let me know if this is ready for review

[viveksahu26]

@viveksahu26 let me know if this is ready for review

Yeah it's ready for review...

[riteshnoronha]

We should not limit ourselves to only GitHub urls. An SBOM could be on any URL, we should just check if this path is a url and download the file and run sbomqs on it. This is adding too much complexity.

[viveksahu26]

I hope you mean downloading the file in memory instead of local storage.

[viveksahu26]

Hey @riteshnoronha , We can't download file from any general URL in using same steps or method. For each general URL one need to construct download URL of that and it differs from server to server. For Example to download git URL , one need to construct raw content URL of it, i.e. https://raw.githubusercontent.com.
Normal URL: https://github.com/interlynk-io/sbomqs/blob/main/samples/sbomqs-spdx-syft.json
Download URL: https://raw.githubusercontent.com/interlynk-io/sbomqs/main/samples/sbomqs-spdx-syft.json

Similarly, if url is google drive url, then you need to construct download URL for GDrive.
Normal URL: https://drive.google.com/file/d/1pL6ykrWZAIxck15tz_f2Rv2Xl3jBNSk8/view?usp=sharing
Download URL: https://drive.google.com/uc?export=download&id=1pL6ykrWZAIxck15tz_f2Rv2Xl3jBNSk8

Similalry for GitLab:
Normal URL: https://gitlab.com/viveksahu26/go-url/-/blob/main/other.spdx.txt?ref_type=heads
Download URL: https://gitlab.com/viveksahu26/go-url/-/raw/main/other.spdx.txt?ref_type=heads

So, one need to know the url representing to servers.

So, it's better for now add a support specifically for git URL and later on if raise issue related to any other URLs then we can deal with that.

[viveksahu26]

Hey @riteshnoronha , remove complexity which were there due to 2 reasons:

• handling response in bit different way
• support for directory, for example: https://github.com/interlynk-io/sbomqs/tree/main/samples. So, now removed support for dir, only support for file url is there: https://github.com/interlynk-io/sbomqs/blob/main/samples/sbomqs-spdx-syft.json

[riteshnoronha]

• Some extra debug information is being printed.
• https://repo1.maven.org/maven2/io/dropwizard/dropwizard-core/4.0.1/dropwizard-core-4.0.1-cyclonedx.json does not work.

If we support urls then this should work.

[riteshnoronha]

Should work in all places we support file names. e.g compliance etc