Add a PkgEvalJob based on NewPkgEval #64
Conversation
maleadt
force-pushed
the
tb/pkgeval
branch
4 times, most recently
from
980cf2d to
3ea3c99
Compare
|
Looks like the tests are failing due to GitHub rate limiting. Is that something you've encountered when running the job? Could be the difference between authenticated and unauthenticated requests, since GitHub permits a higher rate limit for the former. |
|
I've encountered that while working on NewPkgEval, and moved away from Github API usage there. Maybe here we could add the Nanosoldier token to the CI secret variables, and use GitHub OATH? |
|
If the GitHub API isn't being used for NewPkgEval, does that mean it's the existing tests that are hitting rate limits? 🤔 |
|
Yes, the tests use an Github.AnonymousAuth and test submissions strings which look up a branch using the GitHub API. |
|
This is looking good to me. I assume you've done due diligence for the changes, so just let me know when it's ready and I'll merge it, then we can discuss how to get things properly set up for this to go live. |
It would be nice if these separators matched Pkg's syntax though.
- generate a latest symlink to simplify lookup - put the date in the db.json - only parse the db.json, not the actual results (which we don't use) - simplify report generation since comparison jobs always have an against build
|
This has been active for a while now, so I propose merging the PR so that follow-up changes can be reviewed in isolation again. IIUC none of the bots are automatically pulling from master anyway, so this shouldn't really matter until we decide to go an upgrade the cluster. |
|
Benchmarking jobs run from the master branch, but it isn't updated automatically. |
This PR adds a PkgEvalJob to periodically, or on demand, run tests of all packages in General against specific Julia versions. It is build on NewPkgEval, and most of the changes in this PR are confined to
PkgEvalJob.jl.I tried to make the job look like the existing BenchmarkJob as much as possible, so that it is possible to extract common functionality after this PR is merged. This includes the concept of daily builds, the way reports are generated and formatted, including a compressed data dump, etc.
At the same time, PkgEvalJob has some specific dependencies (julia 1.3, NewPkgEval#master, BinaryBuilder#master) and is likely to be under development for a while, so I don't think it's worth to try and integrate this into the existing infrastructure on the MIT cluster. I did some minor refactoring to facilitate running as a separate instance, and I'm currently trial-running this PR on arctic1.
Some example output: maleadt/julia@9babbf5
I also added support for referencing a tag using
#. It would probably be good to unify this syntax with Pkg's though (#for branches,@for versions == tags), but I kept it non-breaking for now. Finally, I added a Project.toml and Manifest, but I'm not sure what the exact set-up on the cluster is.