https://oss-prow.knative.dev/[command-help][plugins] is void of information

[MushuEE]

command-help and plugins must have a config serving problem or other bug leaving them blank.

[MushuEE]

 jsonPayload: {
  component: "deck"   
  error: "error Getting plugin help: Get http://hook:8888/plugin-help: EOF"   
  file: "prow/cmd/deck/main.go:1298"   
  func: "main.handlePluginHelp.func1"   
  handler: "/plugin-help.js"   
  level: "error"   
  msg: "Getting plugin help from hook."   
 }

Looks like issue might be bubbling up from which is configured here

[cjwagner]

I noticed this recently and just looked into it. Hook's /plugin-help endpoint seems to hang before eventually returning a 503 (possibly from k8s?). Subsequent requests can fail immediately with the same status code and EOF message.

The logs suggest that this may be related to excessive time spent listing the repos in some of the served organizations. In particular listing the repos for the GoogleCloudPlatform and google orgs is very slow at about 8 seconds and 18 seconds respectively. It also seems that these requests are made sequentially rather than in parallel.

Since this API is so slow we should try to find an alternative, faster API (perhaps v4 would be better here since we just need the repo names, no additional info), or obviate the requirement for this information.

[MushuEE]

/assign

[MushuEE]

For reference:
curl -I "Accept: application/vnd.github.v3json" "https://api.github.com/orgs/google/repos?per_page=100"

HTTP/2 200 
server: GitHub.com
date: Thu, 18 Mar 2021 22:31:58 GMT
content-type: application/json; charset=utf-8
cache-control: public, max-age=60, s-maxage=60
vary: Accept, Accept-Encoding, Accept, X-Requested-With
etag: W/"b911d5c6da90a7d1958a2062ff311137eacfe51cdf61a7190d267225dbe6aa7c"
x-github-media-type: github.v3; format=json
link: <https://api.github.com/organizations/1342004/repos?per_page=100&page=2>; rel="next", <https://api.github.com/organizations/1342004/repos?per_page=100&page=20>; rel="last"
access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
access-control-allow-origin: *
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'
x-ratelimit-limit: 60
x-ratelimit-remaining: 37
x-ratelimit-reset: 1616109214
x-ratelimit-used: 23
accept-ranges: bytes
x-github-request-id: FEB7:8C27:B99B33:CAA8F3:6053D4DE

For google there are 20 pages. These could be queried for in parallel. per_page=100&page=20>; rel="last"

[MushuEE]

Looks like the solution for pageation here is

query {
    organization(login:"google") {
    repositories(first: 100, after:"Y3Vyc29yOnYyOpHOAVe8uQ==") {
      edges {
        node {
          name
        }
      }
      pageInfo {
        endCursor
        hasNextPage
      }
    }
  }
}

using after:<endCursor to iterate over pages. v4 method returns only what we need.

[MushuEE]

While we can get totalCount there is no way to offset without a cursor to do v4 in parallel

[MushuEE]

Ran some benchmarks:

"google"

---

Executed in 7.89 secs fish external
usr time 1.36 secs 94.00 micros 1.36 secs
sys time 0.93 secs 545.00 micros 0.93 secs

"GoogleCloudPlatform"

---

Executed in 3.95 secs fish external
usr time 1.25 secs 108.00 micros 1.25 secs
sys time 0.90 secs 667.00 micros 0.90 secs