Improve GitHub repository enumeration with filtering mechanisms

[bradlarsen]

A few commands enumerate GitHub repositories:

• scan --github-user=USER
• scan --github-org=ORG
• github repos list --user=USER
• github repos list --org=ORG

These currently do not offer any filtering mechanism on the set of resulting repo URLs. It would be useful to have an option to ignore forked repos.

[bradlarsen]

It would also be useful to have some kind of functionality when enumerating a Git repository to focus only on commits authored by a particular user (This is related to #29). The idea there being that if you had that mechanism, you could still scan forked repos, but only focus on certain commits within the fork.

(The use case in this comment is more complicated than simply skipping forked repos.)

[bradlarsen]

Also useful things:

• Add an option to enumerate repositories belonging to the authenticated user when using a token
• Add an option to enumerate gist repos for particular users
• Add an option to enumerate wiki repos for each project

[CameronLonsdale]

Visibility is also a really useful filtering option. If my token gives access to an entire organisation (since that gives me a higher rate limit from GitHub), can I then specify to only scan public repositories

[bradlarsen]

Yes, an option like --type={all,public,private,forks,sources,member} when using --github-organization=ORG would certainly be useful.

I got those values from the GitHub API docs: https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-organization-repositories

The allowed values are different when listing a regular user's repositories, where you have --type={all,owner,member}: https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-repositories-for-a-user

The allowed values are different yet again when listing the authenticated user's repositories. There you have --type={all,owner,public,private,member}. https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-repositories-for-the-authenticated-user

Hmm, it's a bit tricky to come up with a sensible command-line interface to expose the full set of options.

[CameronLonsdale]

Might make logical sense to split it up with --github-organization and --github-user scanning options. Then you could have a subsequent validation step depending on whether the user matches the auth token you receive, or just wait for the API request to fail with a mismatched type and bubble that back to the terminal.

[CameronLonsdale]

After looking at the github CLI , its quite nice and does all of this already https://cli.github.com/. Maybe you just follow the unix philosophy and if we want to do repo filtering, do it first using github cli and then parallelise scanning with nosey parker

[bradlarsen]

After looking at the github CLI , its quite nice and does all of this already https://cli.github.com/. Maybe you just follow the unix philosophy and if we want to do repo filtering, do it first using github cli and then parallelise scanning with nosey parker

Yeah, that's a solid idea, and the sort of thing I often recommend to people when they run into a situation where Nosey Parker's built-in enumeration capabilities don't hold up: gather whatever content you want in advance, and then scan that location on the filesystem.

That said, I still intend to keep GitHub enumeration capabilities in Nosey Parker. Having something built-in that works much of the time is a big convenience. (When I first implemented this support, I was surprised myself with how useful it was at casting a larger net.)

[bradlarsen]

Note that the official GitHub CLI (gh) has a repos list command, which uses the --source and --fork arguments to specify only non-forks and forks, respectively.