Race conditions

[mbauman]

It appears as though the exact versions that get compared depend upon the time they get started. This can lead to unrelated commits getting involved in the comparison. Based purely upon the behavior I've seen in the reported logs, it appears as though the following things occur:

• Job gets submitted
• Nanosoldier begins building master, grabbing the tip of the master at the time the build starts
• Nanosoldier begins building the merge of the PR, grabbing the tip of the master and the tip of the PR branch at the time this build starts

This can lead to the PR build having extra, unexpected commits in its history, both from the master and PR branch (although the latter can actually work in our favor given the long lead time). Ideally, we'd just figure out the master and the PR merge hashes immediately upon submission to avoid these problems.

[vchuravy]

Hm cursory glancing at the code I think the race condition comes from

Nanosoldier.jl/src/build.jl

Line 34 in 9f04e2f

run(`git fetch --quiet origin +refs/pull/$(pr)/merge:`)

and

Nanosoldier.jl/src/jobs/BenchmarkJob.jl

Line 62 in 9f04e2f

againstbuild = branchref(submission.config, againstrepo, againstbranch)

The latter is setting the the versus to the at the time current master sha, and the former is getting the merge commit at a later point.

[jrevels]

Yeah, the git fetches in build_julia! just need to be moved into the BenchmarkJob constructor

[maleadt]

Fixed in #134; the BenchmarkJob constructor now determines what to test, and build.jl only checks out the configured sha.