Reduce number of requests to pypi server

[B-Souty]

Hi,

We're hosting a private pypi server (pypicloud) and whenever Renovate runs, it hammers it and our server throws a bunch of 503 error.

Some projects are using the same dependencies, yet within a single run, Renovate will make a request for the same dependency for each project.

For example if 2 projects require pandas, Renovate will query pypi twice for pandas in a single run.

I was under the impression dependency versions would be cached when first encountered during a run and subsequent projects using a same dependency would be served from the cache.

Am I doing something wrong or am I misunderstanding how caching works?

---

• Renovate version: 24.29.0 (self-hosted)
• Pypicloud version: 1.1.7

[rarkins]

There should be caching, but I haven't verified it manually for a long time. A test would be:

• Set up two repos both with a requirements.txt and the same single requirement
• Run renovate on both in one run, see if it makes twice as many requests as expected

[viceice]

@B-Souty is there some auth required for your pypi server?

[B-Souty]

That is what I did to test and I see multiple requests for a single package on my pypi server.

repo 1 requirement.txt:

--extra-index-url https://my.pypi.server/simple/
requests==2.20.0

repo 2 requirements.txt:

--extra-index-url https://my.pypi.server/simple/
requests==2.20.0
pandas==1.1.5

The logs on my pypi server are as follow:

[pid: 31|app: 0|req: 1/25] 172.17.0.1 () {32 vars in 445 bytes} [Thu Jan 28 14:43:46 2021] GET /simple/requests => generated 126 bytes in 19 msecs (HTTP/1.1 302) 3 headers in 142 bytes (1 switches on core 0)
[pid: 38|app: 0|req: 3/26] 172.17.0.1 () {32 vars in 448 bytes} [Thu Jan 28 14:43:47 2021] GET /simple/requests/ => generated 390 bytes in 15 msecs (HTTP/1.1 200) 2 headers in 80 bytes (1 switches on core 0)
[pid: 39|app: 0|req: 10/27] 172.17.0.1 () {32 vars in 446 bytes} [Thu Jan 28 14:43:59 2021] GET /simple/requests => generated 126 bytes in 5 msecs (HTTP/1.1 302) 3 headers in 142 bytes (1 switches on core 0)
[pid: 39|app: 0|req: 11/28] 172.17.0.1 () {32 vars in 442 bytes} [Thu Jan 28 14:43:59 2021] GET /simple/pandas => generated 124 bytes in 4 msecs (HTTP/1.1 302) 3 headers in 140 bytes (1 switches on core 0)
[pid: 38|app: 0|req: 4/29] 172.17.0.1 () {32 vars in 448 bytes} [Thu Jan 28 14:44:00 2021] GET /simple/requests/ => generated 390 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 80 bytes (1 switches on core 0)
[pid: 38|app: 0|req: 5/30] 172.17.0.1 () {32 vars in 444 bytes} [Thu Jan 28 14:44:00 2021] GET /simple/pandas/ => generated 111 bytes in 8 msecs (HTTP/1.1 302) 3 headers in 127 bytes (1 switches on core 0)

And the debug logs from Renovate are:

DEBUG: http statistics (repository=myrepo/test-renovate)
       "hostStats": [
         "api.github.com, 2 requests, 349ms request average, 0ms queue average",
         "my.pypi.server, 1 request, 689ms request average, 0ms queue average",
         "gitlab.com, 5 requests, 469ms request average, 1ms queue average",
         "pypi.org, 1 request, 204ms request average, 0ms queue average"
       ],
       "totalRequests": 9

DEBUG: http statistics (repository=myrepo/test-renovate2)
       "hostStats": [
         "api.github.com, 2 requests, 403ms request average, 0ms queue average",
         "my.pypi.server, 2 requests, 464ms request average, 0ms queue average",
         "gitlab.com, 7 requests, 445ms request average, 0ms queue average",
         "pypi.org, 2 requests, 269ms request average, 0ms queue average"
       ],
       "totalRequests": 13

[B-Souty]

@B-Souty is there some auth required for your pypi server?

No auth required.

[viceice]

You can use concurrentRequestLimit to reduce parallel requests to your pypi server.

[B-Souty]

I will give that a try in the meantime. Thanks.

[B-Souty]

Hi @rarkins, Have you had a chance to look at this? I'm happy to open an issue with my findings if needed.

Lowering the concurrentRequestLimit setting isn't an ideal solution as we would have to update all the repos we have using Renovate, possibly multiple times. Not very flexible.

[rarkins]

PyPI caching for non-authenticated lookups was added in #8477. Caching is currently hardcoded to 15 minutes

[B-Souty]

I upgraded Renovate to 24.41.0 and the issue has indeed been fixed. We went from 1228 requests per run to our pypi server to only 228. Awesome work 👍