+ +
+
+
+
+
+ Features • + Install • + Usage • + Why ghtracker • + References • +
+ +--- + +`ghtracker` is a tool for tracking dependents repositories and sorting result by Stars ⭐. It has a simple architecture and is optimized for speed. + +## Features + +- Analysis of 'used by' or 'Dependency graph' in Github repositories and sorting result by Stars +- In-depth Search of Key Code in Used Objects +- Support for Output in JSON Format +- Provision of Aesthetically Pleasing and Easy-to-Read Terminal Table Outputs +- Comprehensive Cross-Platform Compatibility +- Efficient HTTP Request Anti-Ban Mechanism + +## Installation + +### From Source + +`ghtracker` requires **go1.20** to install successfully. Run the following command to install the latest version: + +```sh +go install github.com/zer0yu/ghtracker@latest +``` + +### From Release + +Download from [releases](http://github.com/zer0yu/ghtracker/releases/) + +## Usage + +### Help + +```sh +ghtracker -h +``` + +This will display help for the tool. Here are all the switches it supports. + +```sh +CLI tool for tracking dependents repositories and sorting result by Stars + +Usage: + ghtracker [flags] + +Flags: + -d, --description Show description of packages or repositories (performs additional request per repository) + -h, --help help for ghtracker + -m, --minstar int Minimum number of stars (default=5) (default 5) + -f, --output string File to write output to (optional) + -r, --repositories Sort repositories or packages (default packages) + -o, --rows int Number of showing repositories (default=10) (default 10) + -s, --search string search code at dependents (repositories or packages) + -t, --table View mode + -k, --token string GitHub token + -u, --url string URL to process +``` + +### Basic Usage + +#### 1. Retrieves Packages in 'Used by' or 'Dependency graph' by default, and saves the addresses of projects with more than 5 output stars. + +```sh +ghtracker --url https://github.com/AFLplusplus/LibAFL -t +/ ++---------------------------------------+-------+-------------+ +| URL | STARS | DESCRIPTION | ++---------------------------------------+-------+-------------+ +| https://github.com/AFLplusplus/LibAFL | 1.7K | | +| https://github.com/epi052/feroxfuzz | 183 | | +| https://github.com/fkie-cad/butterfly | 40 | | +| https://github.com/z2-2z/peacock | 0 | | ++---------------------------------------+-------+-------------+ +found 8 packages others packages are private +Complete! +``` + +#### 2. Get the Repositories in the 'Used by' or 'Dependency graph', by default it saves the addresses of projects with the first 10. + +```sh +ghtracker --url https://github.com/AFLplusplus/LibAFL -t -r +/ ++--------------------------------------------------------+-------+-------------+ +| URL | STARS | DESCRIPTION | ++--------------------------------------------------------+-------+-------------+ +| https://github.com/AFLplusplus/LibAFL | 1.7K | | +| https://github.com/hardik05/Damn_Vulnerable_C_Program | 604 | | +| https://github.com/fuzzland/ityfuzz | 524 | | +| https://github.com/epi052/feroxfuzz | 183 | | +| https://github.com/epi052/fuzzing-101-solutions | 119 | | +| https://github.com/tlspuffin/tlspuffin | 117 | | +| https://github.com/Agnoctopus/Tartiflette | 90 | | +| https://github.com/fkie-cad/butterfly | 40 | | +| https://github.com/IntelLabs/PreSiFuzz | 38 | | +| https://github.com/RickdeJager/TrackmaniaFuzzer | 32 | | +| https://github.com/AFLplusplus/libafl_paper_artifacts | 17 | | +| https://github.com/novafacing/libc-fuzzer | 12 | | +| https://github.com/vusec/triereme | 11 | | +| https://github.com/bitterbit/fuzzer-qemu | 7 | | +| https://github.com/atredis-jordan/libafl-workshop-blog | 7 | | +| https://github.com/rezer0dai/LibAFL | 6 | | +| https://github.com/jjjutla/Fuzz | 5 | | ++--------------------------------------------------------+-------+-------------+ +found 190 repositories others repositories are private +Complete! +``` + +#### 3. Save the result in a file in Json format + +```sh +ghtracker --url https://github.com/AFLplusplus/LibAFL -t -r --output ./test.json +``` + +#### 4. Search Code Pattern in 'Dependency graph' (Need Github Token) + +```sh +ghtracker --url https://github.com/AFLplusplus/LibAFL --token your_token_value -t --search AFL --output ./test.json +\ +[INF] https://github.com/AFLplusplus/LibAFL/blob/e117b7199ca902d462edc1de1bc0b3cb71c27aff/scripts/afl-persistent-config with 1747 stars +[INF] https://github.com/AFLplusplus/LibAFL/blob/e117b7199ca902d462edc1de1bc0b3cb71c27aff/libafl_cc/src/afl-coverage-pass.cc with 1747 stars +[INF] https://github.com/AFLplusplus/LibAFL/blob/e117b7199ca902d462edc1de1bc0b3cb71c27aff/libafl_targets/src/cmps/observers/aflpp.rs with 1747 stars +[INF] https://github.com/AFLplusplus/LibAFL/blob/e117b7199ca902d462edc1de1bc0b3cb71c27aff/fuzzers/forkserver_libafl_cc/src/bin/libafl_cc.rs with 1747 stars +[INF] https://github.com/AFLplusplus/LibAFL/blob/e117b7199ca902d462edc1de1bc0b3cb71c27aff/fuzzers/libfuzzer_libpng_aflpp_ui/README.md with 1747 stars +... +... +``` + +#### 5. Get all repositories in the 'Dependency graph' + +```sh +ghtracker --url https://github.com/AFLplusplus/LibAFL -t -r -m 0 --output ./test.json +``` + +# Why ghtracker + +1. Gtihub does not support sorting the 'Dependency graph' by the number of Stars from 2019 until now, so ghtracker + was born. Detials in issue in [1537](https://github.com/isaacs/github/issues/1537) +2. GHTOPDEP does not continue to update support, and there are some issues in the community, e.g., [issue](https://github.com/github-tooling/ghtopdep). ghtracker has optimized and improved on this, and has addressed issues in the community. + +# References +- [GHTOPDEP](https://github.com/github-tooling/ghtopdep) diff --git a/cmd/root.go b/cmd/root.go new file mode 100644 index 0000000..d12ca18 --- /dev/null +++ b/cmd/root.go @@ -0,0 +1,54 @@ +package cmd + +import ( + "github.com/projectdiscovery/gologger" + "github.com/zer0yu/ghtracker/pkg/runner" + "os" + + "github.com/spf13/cobra" + "github.com/zer0yu/ghtracker/pkg/options" +) + +var ghoptions = options.GHTopDepOptions{} + +var rootCmd = &cobra.Command{ + Use: "ghtracker", + Short: "CLI tool for tracking dependents repositories and sorting result by Stars", + Long: ``, + Run: GHTracker, +} + +func Execute() { + err := rootCmd.Execute() + if err != nil { + os.Exit(1) + } +} + +func init() { + rootCmd.PersistentFlags().BoolVarP(&ghoptions.Version, "version", "v", false, "Show version of ghtracker") + rootCmd.PersistentFlags().StringVarP(&ghoptions.URL, "url", "u", "", "URL to process") + rootCmd.PersistentFlags().BoolVarP(&ghoptions.Repositories, "repositories", "r", false, + "Sort repositories or packages (default repositories)") + rootCmd.PersistentFlags().BoolVarP(&ghoptions.Table, "table", "t", false, "View mode") + rootCmd.PersistentFlags().BoolVarP(&ghoptions.Description, "description", "d", false, + "Show description of packages or repositories (performs additional request per repository)") + rootCmd.PersistentFlags().IntVarP(&ghoptions.Rows, "rows", "o", 10, "Number of showing repositories (default=10)") + rootCmd.PersistentFlags().IntVarP(&ghoptions.MinStar, "minstar", "m", 5, "Minimum number of stars (default=5)") + rootCmd.PersistentFlags().StringVarP(&ghoptions.Search, "search", "s", "", + "search code at dependents (repositories or packages)") + rootCmd.PersistentFlags().StringVarP(&ghoptions.Token, "token", "k", os.Getenv("GHTOPDEP_TOKEN"), "GitHub token") + rootCmd.PersistentFlags().StringVarP(&ghoptions.OutputFile, "output", "f", "", "File to write output to (optional)") +} + +func GHTracker(_ *cobra.Command, _ []string) { + newRunner, err := runner.NewRunner(&ghoptions) + if err != nil { + gologger.Fatal().Msgf("Could not create runner: %s\n", err) + } + + err = newRunner.RunGHCrawler() + if err != nil { + gologger.Fatal().Msgf("Could not run fuzz engine: %s\n", err) + } +} diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..3e097bb --- /dev/null +++ b/go.mod @@ -0,0 +1,59 @@ +module github.com/zer0yu/ghtracker + +go 1.21.3 + +require ( + github.com/PuerkitoBio/goquery v1.8.1 + github.com/briandowns/spinner v1.23.0 + github.com/gofri/go-github-ratelimit v1.1.0 + github.com/google/go-github/v58 v58.0.0 + github.com/imroc/req/v3 v3.42.3 + github.com/olekukonko/tablewriter v0.0.5 + github.com/projectdiscovery/gologger v1.1.12 + github.com/spf13/cobra v1.8.0 +) + +require ( + github.com/andybalholm/brotli v1.0.6 // indirect + github.com/andybalholm/cascadia v1.3.2 // indirect + github.com/cloudflare/circl v1.3.7 // indirect + github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 // indirect + github.com/fatih/color v1.7.0 // indirect + github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect + github.com/golang/snappy v0.0.4 // indirect + github.com/google/go-querystring v1.1.0 // indirect + github.com/google/pprof v0.0.0-20231229205709-960ae82b1e42 // indirect + github.com/hashicorp/errwrap v1.1.0 // indirect + github.com/hashicorp/go-multierror v1.1.1 // indirect + github.com/inconshreveable/mousetrap v1.1.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/klauspost/compress v1.17.4 // indirect + github.com/klauspost/pgzip v1.2.5 // indirect + github.com/logrusorgru/aurora v2.0.3+incompatible // indirect + github.com/mattn/go-colorable v0.1.2 // indirect + github.com/mattn/go-isatty v0.0.8 // indirect + github.com/mattn/go-runewidth v0.0.9 // indirect + github.com/mholt/archiver/v3 v3.5.1 // indirect + github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/nwaples/rardecode v1.1.3 // indirect + github.com/onsi/ginkgo/v2 v2.13.2 // indirect + github.com/pierrec/lz4/v4 v4.1.2 // indirect + github.com/quic-go/qpack v0.4.0 // indirect + github.com/quic-go/qtls-go1-20 v0.4.1 // indirect + github.com/quic-go/quic-go v0.40.1 // indirect + github.com/refraction-networking/utls v1.6.0 // indirect + github.com/spf13/pflag v1.0.5 // indirect + github.com/ulikunitz/xz v0.5.11 // indirect + github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect + go.uber.org/mock v0.4.0 // indirect + golang.org/x/crypto v0.18.0 // indirect + golang.org/x/exp v0.0.0-20240103183307-be819d1f06fc // indirect + golang.org/x/mod v0.14.0 // indirect + golang.org/x/net v0.20.0 // indirect + golang.org/x/sys v0.16.0 // indirect + golang.org/x/term v0.16.0 // indirect + golang.org/x/text v0.14.0 // indirect + golang.org/x/tools v0.16.1 // indirect + gopkg.in/djherbis/times.v1 v1.3.0 // indirect +) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..16c4634 --- /dev/null +++ b/go.sum @@ -0,0 +1,179 @@ +github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM= +github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ= +github.com/andybalholm/brotli v1.0.1/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y= +github.com/andybalholm/brotli v1.0.6 h1:Yf9fFpf49Zrxb9NlQaluyE92/+X7UVHlhMNJN2sxfOI= +github.com/andybalholm/brotli v1.0.6/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig= +github.com/andybalholm/cascadia v1.3.1/go.mod h1:R4bJ1UQfqADjvDa4P6HZHLh/3OxWWEqc0Sk8XGwHqvA= +github.com/andybalholm/cascadia v1.3.2 h1:3Xi6Dw5lHF15JtdcmAHD3i1+T8plmv7BQ/nsViSLyss= +github.com/andybalholm/cascadia v1.3.2/go.mod h1:7gtRlve5FxPPgIgX36uWBX58OdBsSS6lUvCFb+h7KvU= +github.com/briandowns/spinner v1.23.0 h1:alDF2guRWqa/FOZZYWjlMIx2L6H0wyewPxo/CH4Pt2A= +github.com/briandowns/spinner v1.23.0/go.mod h1:rPG4gmXeN3wQV/TsAY4w8lPdIM6RX3yqeBQJSrbXjuE= +github.com/cloudflare/circl v1.3.7 h1:qlCDlTPz2n9fu58M0Nh1J/JzcFpfgkFHHX3O35r5vcU= +github.com/cloudflare/circl v1.3.7/go.mod h1:sRTcRWXGLrKw6yIGJ+l7amYJFfAXbZG0kBSc8r4zxgA= +github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 h1:iFaUwBSo5Svw6L7HYpRu/0lE3e0BaElwnNO1qkNQxBY= +github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5/go.mod h1:qssHWj60/X5sZFNxpG4HBPDHVqxNm4DfnCKgrbZOT+s= +github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY= +github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys= +github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= +github.com/go-logr/logr v1.3.0 h1:2y3SDp0ZXuc6/cjLSZ+Q3ir+QB9T/iG5yYRXqsagWSY= +github.com/go-logr/logr v1.3.0/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= +github.com/gofri/go-github-ratelimit v1.1.0 h1:ijQ2bcv5pjZXNil5FiwglCg8wc9s8EgjTmNkqjw8nuk= +github.com/gofri/go-github-ratelimit v1.1.0/go.mod h1:OnCi5gV+hAG/LMR7llGhU7yHt44se9sYgKPnafoL7RY= +github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= +github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/golang/snappy v0.0.2/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= +github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-github/v58 v58.0.0 h1:Una7GGERlF/37XfkPwpzYJe0Vp4dt2k1kCjlxwjIvzw= +github.com/google/go-github/v58 v58.0.0/go.mod h1:k4hxDKEfoWpSqFlc8LTpGd9fu2KrV1YAa6Hi6FmDNY4= +github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8= +github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/pprof v0.0.0-20231229205709-960ae82b1e42 h1:dHLYa5D8/Ta0aLR2XcPsrkpAgGeFs6thhMcQK0oQ0n8= +github.com/google/pprof v0.0.0-20231229205709-960ae82b1e42/go.mod h1:czg5+yv1E0ZGTi6S6vVK1mke0fV+FaUhNGcd6VRS9Ik= +github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= +github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= +github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= +github.com/imroc/req/v3 v3.42.3 h1:ryPG2AiwouutAopwPxKpWKyxgvO8fB3hts4JXlh3PaE= +github.com/imroc/req/v3 v3.42.3/go.mod h1:Axz9Y/a2b++w5/Jht3IhQsdBzrG1ftJd1OJhu21bB2Q= +github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= +github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A= +github.com/klauspost/compress v1.11.4/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= +github.com/klauspost/compress v1.17.4 h1:Ej5ixsIri7BrIjBkRZLTo6ghwrEtHFk7ijlczPW4fZ4= +github.com/klauspost/compress v1.17.4/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM= +github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= +github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE= +github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= +github.com/logrusorgru/aurora v2.0.3+incompatible h1:tOpm7WcpBTn4fjmVfgpQq0EfczGlG91VSDkswnjF5A8= +github.com/logrusorgru/aurora v2.0.3+incompatible/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4= +github.com/mattn/go-colorable v0.1.2 h1:/bC9yWikZXAL9uJdulbSfyVNIR3n3trXl+v8+1sx8mU= +github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= +github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE= +github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= +github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0= +github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI= +github.com/mholt/archiver/v3 v3.5.1 h1:rDjOBX9JSF5BvoJGvjqK479aL70qh9DIpZCl+k7Clwo= +github.com/mholt/archiver/v3 v3.5.1/go.mod h1:e3dqJ7H78uzsRSEACH1joayhuSyhnonssnDhppzS1L4= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 h1:ZqeYNhU3OHLH3mGKHDcjJRFFRrJa6eAM5H+CtDdOsPc= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/nwaples/rardecode v1.1.0/go.mod h1:5DzqNKiOdpKKBH87u8VlvAnPZMXcGRhxWkRpHbbfGS0= +github.com/nwaples/rardecode v1.1.3 h1:cWCaZwfM5H7nAD6PyEdcVnczzV8i/JtotnyW/dD9lEc= +github.com/nwaples/rardecode v1.1.3/go.mod h1:5DzqNKiOdpKKBH87u8VlvAnPZMXcGRhxWkRpHbbfGS0= +github.com/olekukonko/tablewriter v0.0.5 h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec= +github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY= +github.com/onsi/ginkgo/v2 v2.13.2 h1:Bi2gGVkfn6gQcjNjZJVO8Gf0FHzMPf2phUei9tejVMs= +github.com/onsi/ginkgo/v2 v2.13.2/go.mod h1:XStQ8QcGwLyF4HdfcZB8SFOS/MWCgDuXMSBe6zrvLgM= +github.com/onsi/gomega v1.29.0 h1:KIA/t2t5UBzoirT4H9tsML45GEbo3ouUnBHsCfD2tVg= +github.com/onsi/gomega v1.29.0/go.mod h1:9sxs+SwGrKI0+PWe4Fxa9tFQQBG5xSsSbMXOI8PPpoQ= +github.com/pierrec/lz4/v4 v4.1.2 h1:qvY3YFXRQE/XB8MlLzJH7mSzBs74eA2gg52YTk6jUPM= +github.com/pierrec/lz4/v4 v4.1.2/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/projectdiscovery/gologger v1.1.12 h1:uX/QkQdip4PubJjjG0+uk5DtyAi1ANPJUvpmimXqv4A= +github.com/projectdiscovery/gologger v1.1.12/go.mod h1:DI8nywPLERS5mo8QEA9E7gd5HZ3Je14SjJBH3F5/kLw= +github.com/quic-go/qpack v0.4.0 h1:Cr9BXA1sQS2SmDUWjSofMPNKmvF6IiIfDRmgU0w1ZCo= +github.com/quic-go/qpack v0.4.0/go.mod h1:UZVnYIfi5GRk+zI9UMaCPsmZ2xKJP7XBUvVyT1Knj9A= +github.com/quic-go/qtls-go1-20 v0.4.1 h1:D33340mCNDAIKBqXuAvexTNMUByrYmFYVfKfDN5nfFs= +github.com/quic-go/qtls-go1-20 v0.4.1/go.mod h1:X9Nh97ZL80Z+bX/gUXMbipO6OxdiDi58b/fMC9mAL+k= +github.com/quic-go/quic-go v0.40.1 h1:X3AGzUNFs0jVuO3esAGnTfvdgvL4fq655WaOi1snv1Q= +github.com/quic-go/quic-go v0.40.1/go.mod h1:PeN7kuVJ4xZbxSv/4OX6S1USOX8MJvydwpTx31vx60c= +github.com/refraction-networking/utls v1.6.0 h1:X5vQMqVx7dY7ehxxqkFER/W6DSjy8TMqSItXm8hRDYQ= +github.com/refraction-networking/utls v1.6.0/go.mod h1:kHJ6R9DFFA0WsRgBM35iiDku4O7AqPR6y79iuzW7b10= +github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= +github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= +github.com/ulikunitz/xz v0.5.9/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= +github.com/ulikunitz/xz v0.5.11 h1:kpFauv27b6ynzBNT/Xy+1k+fK4WswhN/6PN5WhFAGw8= +github.com/ulikunitz/xz v0.5.11/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= +github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo= +github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= +go.uber.org/mock v0.4.0 h1:VcM4ZOtdbR4f6VXfiOpwpVJDL6lCReaZ6mw31wqh7KU= +go.uber.org/mock v0.4.0/go.mod h1:a6FSlNadKUHUa9IP5Vyt1zh4fC7uAwxMutEAscFbkZc= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc= +golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= +golang.org/x/exp v0.0.0-20240103183307-be819d1f06fc h1:ao2WRsKSzW6KuUY9IWPwWahcHCgR0s52IfwutMfEbdM= +golang.org/x/exp v0.0.0-20240103183307-be819d1f06fc/go.mod h1:iRJReGqOEeBhDZGkGbynYwcHlctCvnjTYIamk7uXpHI= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0= +golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= +golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= +golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo= +golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU= +golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= +golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= +golang.org/x/term v0.16.0 h1:m+B6fahuftsE9qjo0VWp2FW0mB3MTJvR0BaMQrq0pmE= +golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= +golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= +golang.org/x/tools v0.16.1 h1:TLyB3WofjdOEepBHAU20JdNC1Zbg87elYofWYAY5oZA= +golang.org/x/tools v0.16.1/go.mod h1:kYVVN6I1mBNoB1OX+noeBjbRk4IUEPa7JJ+TJMEooJ0= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw= +google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/djherbis/times.v1 v1.3.0 h1:uxMS4iMtH6Pwsxog094W0FYldiNnfY/xba00vq6C2+o= +gopkg.in/djherbis/times.v1 v1.3.0/go.mod h1:AQlg6unIsrsCEdQYhTzERy542dz6SFdQFZFv6mUY0P8= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/main.go b/main.go new file mode 100644 index 0000000..9b3ee1b --- /dev/null +++ b/main.go @@ -0,0 +1,7 @@ +package main + +import "github.com/zer0yu/ghtracker/cmd" + +func main() { + cmd.Execute() +} diff --git a/pkg/config/config.go b/pkg/config/config.go new file mode 100644 index 0000000..8a06d8a --- /dev/null +++ b/pkg/config/config.go @@ -0,0 +1,9 @@ +package config + +const ( + NextButtonSelector = "#dependents > div.paginate-container > div > a" + ItemSelector = "#dependents > div.Box > div.flex-items-center" + RepoSelector = "span > a.text-bold" + StarsSelector = "div > span:nth-child(1)" + GithubURL = "https://github.com" +) diff --git a/pkg/options/options.go b/pkg/options/options.go new file mode 100644 index 0000000..08a3ee8 --- /dev/null +++ b/pkg/options/options.go @@ -0,0 +1,14 @@ +package options + +type GHTopDepOptions struct { + Version bool // Version is a flag that prints the current version. + URL string // URL is the URL to fetch repositories from. + Repositories bool // Repositories is a flag that indicates if repositories should be fetched. + Table bool // Table is a flag that indicates if the output should be in table format. + Description bool // Description is a flag that indicates if the description of the repository should be fetched. + Rows int // Rows is the number of rows to show. + MinStar int // MinStar is the minimum number of stars to show. + Search string // Search is the search code at dependents (repositories/packages). + Token string // Token is the GitHub token to use for authenticated requests. + OutputFile string // Output is the file to write founds to. +} diff --git a/pkg/runner/banners.go b/pkg/runner/banners.go new file mode 100644 index 0000000..f3c0a6e --- /dev/null +++ b/pkg/runner/banners.go @@ -0,0 +1,21 @@ +package runner + +import "github.com/projectdiscovery/gologger" + +const banner = ` + __ __ __ + ____ / /_ / /__________ ______/ /_____ _____ + / __ \/ __ \/ __/ ___/ __ \/ ___/ //_/ _ \/ ___/ + / /_/ / / / / /_/ / / /_/ / /__/ ,< / __/ / + \__, /_/ /_/\__/_/ \__,_/\___/_/|_|\___/_/ +/____/ +` + +// Name +const ToolName = `ghtracker` + +// showBanner is used to show the banner to the user +func showBanner() { + gologger.Print().Msgf("%s\n", banner) + gologger.Print().Msgf("\t\tVersion v1.0.0\n\n") +} diff --git a/pkg/runner/engine.go b/pkg/runner/engine.go new file mode 100644 index 0000000..8579bb1 --- /dev/null +++ b/pkg/runner/engine.go @@ -0,0 +1,267 @@ +package runner + +import ( + "context" + "encoding/json" + "fmt" + "github.com/PuerkitoBio/goquery" + "github.com/google/go-github/v58/github" + "github.com/olekukonko/tablewriter" + "github.com/projectdiscovery/gologger" + "github.com/zer0yu/ghtracker/pkg/config" + "github.com/zer0yu/ghtracker/pkg/utils/textwrap" + "io" + "math" + "net/url" + "os" + "sort" + "strconv" + "strings" +) + +func (r *GHRunner) GHCrawlEngine(writers []io.Writer) error { + destination := "repository" + destinations := "repositories" + repositories := r.options.Repositories // Assuming repositories is a list of strings + + if !repositories { + destination = "package" + destinations = "packages" + } + // + var repos []GHRepo + moreThanZeroCount := 0 + totalReposCount := 0 + + pageURL, _ := r.getPageUrl(destination) + //fmt.Println(pageURL) + for { + resp, err := r.reqClient.R().Get(pageURL) + if err != nil { + gologger.Error().Msgf("Req pageURL Error!") + } + doc, _ := goquery.NewDocumentFromReader(resp.Body) + dependents := doc.Find(config.ItemSelector) + totalReposCount += dependents.Length() + + dependents.Each(func(i int, s *goquery.Selection) { + repoStarsList := s.Find(config.StarsSelector) + if repoStarsList.Length() > 0 { + repoStars := strings.Replace(repoStarsList.First().Text(), ",", "", -1) + repoStarsNum, _ := strconv.Atoi(strings.TrimSpace(repoStars)) + + if repoStarsNum != 0 { + moreThanZeroCount++ + } + + if repoStarsNum >= r.options.MinStar { + relativeRepoURL, _ := s.Find(config.RepoSelector).First().Attr("href") + repoURL := fmt.Sprintf("%s%s", config.GithubURL, relativeRepoURL) + + if !alreadyAdded(repoURL, repos) && repoURL != pageURL { + if r.options.Description { + repoDescription, _ := r.fetchDescription(r.ghClient.ghctx, relativeRepoURL) + repos = append(repos, GHRepo{URL: repoURL, Stars: repoStarsNum, + Description: repoDescription}) + } else { + repos = append(repos, GHRepo{URL: repoURL, Stars: repoStarsNum}) + } + } + } + } + }) + + node := doc.Find(config.NextButtonSelector) + if node.Length() == 2 { + pageURL, _ = node.Eq(1).Attr("href") + } else if node.Length() == 0 || node.First().Text() == "Previous" { + break + } else if node.First().Text() == "Next" { + pageURL, _ = node.First().Attr("href") + } + } + + sortedRepos := sortRepos(repos, totalReposCount) + + if r.options.Search != "" { + for _, repo := range repos { + u, _ := url.Parse(repo.URL) + repoPath := strings.TrimPrefix(u.Path, "/") + query := fmt.Sprintf("%s repo:%s", r.options.Search, repoPath) + results, _, _ := r.ghClient.gitClient.Search.Code(r.ghClient.ghctx, query, nil) + + for _, s := range results.CodeResults { + gologger.Info().Msgf("%s with %d stars\n", *s.HTMLURL, repo.Stars) + } + + outputWriter := NewOutputWriter(true) + for _, writer := range writers { + err := outputWriter.writeJSONResults(results.CodeResults, writer) + if err != nil { + gologger.Error().Msgf("Could not write results for %s: %s\n", query, err) + } + } + + } + } else { + r.showResult(sortedRepos, totalReposCount, moreThanZeroCount, destinations, writers) + } + + return nil +} + +type GHPackage struct { + Count int + PackageID string +} + +type GHRepo struct { + URL string + Stars int + Description string +} + +type GHRepo4Show struct { + URL string + Stars string + Description string +} + +func (r *GHRunner) getPageUrl(destination string) (string, error) { + pageURL := fmt.Sprintf("%s/network/dependents?dependent_type=%s", r.options.URL, strings.ToUpper(destination)) + resp, err := r.reqClient.R().Get(pageURL) + if err != nil { + gologger.Error().Msgf("Req pageURL Error!") + } + doc, err := goquery.NewDocumentFromReader(resp.Body) + if err != nil { // Append raw dump content to error message if goquery parse failed to help troubleshoot. + gologger.Error().Msgf("failed to parse html: %s, raw content:\n%s", err.Error(), resp.Dump()) + } + link := doc.Find(".select-menu-item") + if link.Length() > 0 { + packages := make([]GHPackage, 0) + link.Each(func(i int, s *goquery.Selection) { + href, _ := s.Attr("href") + repoURL := fmt.Sprintf("https://github.com/%s", href) + resp, err := r.reqClient.R().Get(repoURL) + if err != nil { + // 处理错误 + gologger.Error().Msgf("Req repoURL Error!") + } + parsedItem, err := goquery.NewDocumentFromReader(resp.Body) + if err != nil { + gologger.Error().Msgf("failed to parse html: %s, raw content:\n%s", err.Error(), resp.Dump()) + } + hrefURL, _ := url.Parse(repoURL) + packageID := strings.Split(hrefURL.RawQuery, "=")[1] + + countStr := parsedItem.Find(".table-list-filters a:first-child").First().Text() + countStr = strings.Split(countStr, " ")[0] + countStr = strings.ReplaceAll(countStr, ",", "") + count, _ := strconv.Atoi(countStr) + + packages = append(packages, GHPackage{ + Count: count, + PackageID: packageID, + }) + }) + sort.Slice(packages, func(i, j int) bool { + return packages[i].Count > packages[j].Count + }) + + mostPopularPackageID := packages[0].PackageID + pageURL = fmt.Sprintf("%s/network/dependents?dependent_type=%s&package_id=%s", r.options.URL, + strings.ToUpper(destination), mostPopularPackageID) + } + return pageURL, nil +} + +// fetchDescription fetches the description of a repository +func (r *GHRunner) fetchDescription(ctx context.Context, relativeURL string) (string, error) { + parts := strings.Split(relativeURL, "/") + if len(parts) < 2 { + return "", fmt.Errorf("invalid relative URL") + } + owner, repo := parts[0], parts[1] + repoinfo, _, err := r.ghClient.gitClient.Repositories.Get(ctx, owner, repo) + if _, ok := err.(*github.RateLimitError); ok { + gologger.Error().Msgf("hit rate limit") + } + //fmt.Println(repoinfo.GetDescription()) + if repoinfo.GetDescription() != "" { + return textwrap.Shorten(repoinfo.GetDescription(), 60), nil + } + + return " ", nil +} + +func alreadyAdded(repoURL string, repos []GHRepo) bool { + for _, repo := range repos { + if repo.URL == repoURL { + return true + } + } + return false +} + +func sortRepos(repos []GHRepo, rows int) []GHRepo { + sort.Slice(repos, func(i, j int) bool { + return repos[i].Stars > repos[j].Stars + }) + + if rows > len(repos) { + rows = len(repos) + } + return repos[:rows] +} + +// showResult shows the result of the search or fmt the result in table format +func (r *GHRunner) showResult(repos []GHRepo, totalReposCount int, moreThanZeroCount int, destinations string, writers []io.Writer) { + if r.options.Table { + if len(repos) > 0 { + repos4show := readableStars(repos) + tw := tablewriter.NewWriter(os.Stdout) + tw.SetHeader([]string{"URL", "Stars", "Description"}) + for _, repo := range repos4show { + tw.Append([]string{repo.URL, repo.Stars, repo.Description}) + } + tw.Render() + fmt.Printf("found %d %s others %s are private\n", totalReposCount, destinations, destinations) + fmt.Printf("found %d %s with more than zero star\n", moreThanZeroCount, destinations) + } else { + fmt.Printf("Doesn't find any %s that match search request\n", destinations) + } + } else { + reposJSON, _ := json.Marshal(readableStars(repos)) + fmt.Println(string(reposJSON)) + } + + outputWriter := NewOutputWriter(true) + for _, writer := range writers { + err := outputWriter.writeJSONResults(readableStars(repos), writer) + if err != nil { + gologger.Error().Msgf("Could not write results for %s: %s\n", r.options.OutputFile, err) + } + } + +} + +func humanize(num int) string { + if num < 1000 { + return fmt.Sprintf("%d", num) + } else if num < 10000 { + return fmt.Sprintf("%.1fK", math.Round(float64(num)/100.0)/10.0) + } else if num < 1000000 { + return fmt.Sprintf("%.0fK", math.Round(float64(num)/1000.0)) + } else { + return fmt.Sprintf("%d", num) + } +} + +func readableStars(repos []GHRepo) []GHRepo4Show { + repos4show := make([]GHRepo4Show, 0) + for i := range repos { + repos4show = append(repos4show, GHRepo4Show{URL: repos[i].URL, Stars: humanize(repos[i].Stars), Description: repos[i].Description}) + } + return repos4show +} diff --git a/pkg/runner/engine_test.go b/pkg/runner/engine_test.go new file mode 100644 index 0000000..3a46afb --- /dev/null +++ b/pkg/runner/engine_test.go @@ -0,0 +1,58 @@ +package runner + +import ( + "testing" +) + +func TestAlreadyAdded(t *testing.T) { + repos := []GHRepo{ + {URL: "https://github.com/zer0yu/ghtopdep"}, + {URL: "https://github.com/projectdiscovery/gologger"}, + } + + if !alreadyAdded("https://github.com/zer0yu/ghtopdep", repos) { + t.Errorf("alreadyAdded function failed, expected %v, got %v", true, false) + } + + if alreadyAdded("https://github.com/nonexistent/repo", repos) { + t.Errorf("alreadyAdded function failed, expected %v, got %v", false, true) + } +} + +func TestSortRepos(t *testing.T) { + repos := []GHRepo{ + {URL: "https://github.com/zer0yu/ghtopdep", Stars: 10}, + {URL: "https://github.com/projectdiscovery/gologger", Stars: 20}, + } + + sortedRepos := sortRepos(repos, 2) + + if sortedRepos[0].Stars != 20 { + t.Errorf("sortRepos function failed, expected %v, got %v", 20, sortedRepos[0].Stars) + } + + if sortedRepos[1].Stars != 10 { + t.Errorf("sortRepos function failed, expected %v, got %v", 10, sortedRepos[1].Stars) + } +} + +func TestReadableStars(t *testing.T) { + repos := []GHRepo{ + {Stars: 999}, + {Stars: 1000}, + {Stars: 9999}, + {Stars: 10000}, + {Stars: 999999}, + {Stars: 1000000}, + } + + expected := []string{"999", "1.0K", "10.0K", "10K", "1000K", "1000000"} + + repos4show := readableStars(repos) + + for i, repo := range repos4show { + if repo.Stars != expected[i] { + t.Errorf("readableStars function failed, expected %v, got %v", expected[i], repo.Stars) + } + } +} diff --git a/pkg/runner/outputer.go b/pkg/runner/outputer.go new file mode 100644 index 0000000..4f2cf08 --- /dev/null +++ b/pkg/runner/outputer.go @@ -0,0 +1,58 @@ +package runner + +import ( + "errors" + jsoniter "github.com/json-iterator/go" + "io" + "os" + "path/filepath" +) + +// OutputWriter outputs content to writers. +type OutputWriter struct { + JSON bool +} + +// NewOutputWriter creates a new OutputWriter +func NewOutputWriter(json bool) *OutputWriter { + return &OutputWriter{JSON: json} +} + +func (o *OutputWriter) createFile(filename string, appendToFile bool) (*os.File, error) { + if filename == "" { + return nil, errors.New("empty filename") + } + + dir := filepath.Dir(filename) + + if dir != "" { + if _, err := os.Stat(dir); os.IsNotExist(err) { + err := os.MkdirAll(dir, os.ModePerm) + if err != nil { + return nil, err + } + } + } + + var file *os.File + var err error + if appendToFile { + file, err = os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) + } else { + file, err = os.Create(filename) + } + if err != nil { + return nil, err + } + + return file, nil +} + +func (o *OutputWriter) writeJSONResults(results interface{}, writer io.Writer) error { + encoder := jsoniter.NewEncoder(writer) + err := encoder.Encode(results) + if err != nil { + return err + } + return nil +} diff --git a/pkg/runner/runner.go b/pkg/runner/runner.go new file mode 100644 index 0000000..0da1dfc --- /dev/null +++ b/pkg/runner/runner.go @@ -0,0 +1,121 @@ +package runner + +import ( + "context" + "fmt" + "github.com/briandowns/spinner" + "github.com/gofri/go-github-ratelimit/github_ratelimit" + "github.com/google/go-github/v58/github" + "github.com/imroc/req/v3" + "github.com/projectdiscovery/gologger" + "github.com/zer0yu/ghtracker/pkg/options" + "io" + "os" + "time" +) + +type GHRunner struct { + options *options.GHTopDepOptions + ghClient *GHClient + reqClient *req.Client +} + +type GHClient struct { + gitClient *github.Client + ghctx context.Context +} + +func NewRunner(options *options.GHTopDepOptions) (*GHRunner, error) { + ghRunner := &GHRunner{options: options} + if options.Version { + showBanner() + os.Exit(0) + } + + if options.URL == "" { + gologger.Fatal().Msgf("URL is empty!") + } + + if (options.Description || options.Search != "") && options.Token != "" { + rateLimiter, err := github_ratelimit.NewRateLimitWaiterClient(nil) + if err != nil { + gologger.Error().Msgf("Init rateLimiter Error: %v\n", err) + } + + // Set GitHub authentication information and configure the rate limit + ghRunner.ghClient = &GHClient{ + gitClient: nil, + ghctx: context.Background(), + } + + ghRunner.ghClient.gitClient = github.NewClient(rateLimiter).WithAuthToken(options.Token) + + // verify token + _, resp, err := ghRunner.ghClient.gitClient.Users.Get(ghRunner.ghClient.ghctx, "") + if err != nil { + //fmt.Printf("\nerror: %v\n", err) + gologger.Error().Msgf("Token is invalid!") + } + + // If a Token Expiration has been set, it will be displayed. + if !resp.TokenExpiration.IsZero() { + gologger.Error().Msgf("Token Expiration: %v\n", resp.TokenExpiration) + } + } else if (options.Description || options.Search != "") && options.Token == "" { + gologger.Error().Msgf("Please provide token!") + } + + // set req client + ghRunner.reqClient = req.C(). + EnableDumpEachRequest(). + OnAfterResponse(func(client *req.Client, resp *req.Response) error { + if resp.Err != nil { // Ignore when there is an underlying error, e.g. network error. + return nil + } + // Treat non-successful responses as errors, record raw dump content in error message. + if !resp.IsSuccessState() { // Status code is not between 200 and 299. + resp.Err = fmt.Errorf("bad response, raw content:\n%s", resp.Dump()) + } + return nil + }) + ghRunner.reqClient.ImpersonateChrome() + ghRunner.reqClient.R(). + SetRetryCount(15). + SetRetryBackoffInterval(10*time.Second, 20*time.Second). + SetRetryFixedInterval(2 * time.Second) + + return ghRunner, nil +} + +func (r *GHRunner) RunGHCrawler() error { + s := spinner.New(spinner.CharSets[9], 100*time.Millisecond) + s.Suffix = "\n" + s.FinalMSG = "Complete!\n" + s.Start() + outputs := []io.Writer{} + if r.options.OutputFile != "" { + outputWriter := NewOutputWriter(true) + file, err := outputWriter.createFile(r.options.OutputFile, true) + if err != nil { + gologger.Error().Msgf("Could not create file %s: %s\n", r.options.OutputFile, err) + return err + } + err = r.GHCrawlEngine(append(outputs, file)) + if err != nil { + gologger.Error().Msgf("Run GHCrawlEngine Error!") + s.Stop() + return err + } + file.Close() + } else { + err := r.GHCrawlEngine(outputs) + if err != nil { + gologger.Error().Msgf("Run GHCrawlEngine Error!") + s.Stop() + return err + } + } + + s.Stop() + return nil +} diff --git a/pkg/utils/textwrap/textwrap.go b/pkg/utils/textwrap/textwrap.go new file mode 100644 index 0000000..f8cbd3b --- /dev/null +++ b/pkg/utils/textwrap/textwrap.go @@ -0,0 +1,22 @@ +package textwrap + +import "unicode/utf8" + +func Shorten(text string, maxLen int) string { + lastSpaceIx := maxLen + len := 0 + for i, r := range text { + if utf8.RuneLen(r) > 1 { + len += 2 + } else { + len++ + } + if len > maxLen { + return text[:lastSpaceIx] + "..." + } + if r == ' ' { + lastSpaceIx = i + } + } + return text +} diff --git a/pkg/utils/textwrap/textwrap_test.go b/pkg/utils/textwrap/textwrap_test.go new file mode 100644 index 0000000..103b728 --- /dev/null +++ b/pkg/utils/textwrap/textwrap_test.go @@ -0,0 +1,22 @@ +package textwrap + +import "testing" + +func TestShorten(t *testing.T) { + testCases := []struct { + text string + maxLen int + expected string + }{ + {"Hello, world!", 10, "Hello,..."}, + {"Hello, world!", 5, "Hello..."}, + {"Hello", 10, "Hello"}, + } + + for _, tc := range testCases { + result := Shorten(tc.text, tc.maxLen) + if result != tc.expected { + t.Errorf("Shorten(%q, %d) = %q; want %q", tc.text, tc.maxLen, result, tc.expected) + } + } +} diff --git a/pkg/utils/url/url.go b/pkg/utils/url/url.go new file mode 100644 index 0000000..0e049c0 --- /dev/null +++ b/pkg/utils/url/url.go @@ -0,0 +1,24 @@ +package url + +import ( + "net/url" + "path" + "strings" +) + +func GetRelativeURL(inputURL string) (string, error) { + if !strings.HasPrefix(inputURL, "http://") && !strings.HasPrefix(inputURL, "https://") { + inputURL = "https://" + inputURL + } + + u, err := url.Parse(inputURL) + if err != nil { + return "", err + } + + // 清理路径并删除开头的斜杠 + relativeURL := path.Clean(u.Path) + relativeURL = strings.TrimPrefix(relativeURL, "/") + + return relativeURL, nil +} diff --git a/pkg/utils/url/url_test.go b/pkg/utils/url/url_test.go new file mode 100644 index 0000000..e0cb4f5 --- /dev/null +++ b/pkg/utils/url/url_test.go @@ -0,0 +1,26 @@ +package url + +import ( + "testing" +) + +func TestGetRelativeURL(t *testing.T) { + testCases := []struct { + input string + expected string + }{ + {"https://github.com/AFLplusplus/LibAFL", "AFLplusplus/LibAFL"}, + {"https://github.com/AFLplusplus/LibAFL/", "AFLplusplus/LibAFL"}, + {"github.com/AFLplusplus/LibAFL", "AFLplusplus/LibAFL"}, + } + + for _, tc := range testCases { + result, err := GetRelativeURL(tc.input) + if err != nil { + t.Errorf("GetRelativeURL(%q) returned error: %v", tc.input, err) + } + if result != tc.expected { + t.Errorf("GetRelativeURL(%q) = %q; want %q", tc.input, result, tc.expected) + } + } +} diff --git a/static/ghtracker-logo.png b/static/ghtracker-logo.png new file mode 100644 index 0000000..0f825f2 Binary files /dev/null and b/static/ghtracker-logo.png differ