-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Total regression in codecov coverage report looks suspicious #35759
Comments
#35670 (comment) is probably relevant. cc @bcoe |
@puzpuzpuz @richardlau I'm not 100% sure the reason, but codecov.io seems to get a bit confused about the order that commits land to our main branch: ☝️ it's expected that coverage will be about 87% now, but it doesn't look like codecov.io has caught up yet. I believe the notifications will stabilize once it catches up. |
@bcoe thanks for the clarification. It makes sense. I'm going to hide the codecov message and close this issue. Feel free to reopen it if you think it's necessary. |
see also: #35779 |
@thomasrockhu codecov's comments on PRs seem to behave somewhat strangely on the Node.js project: The HEAD of our default branch has been at 87% coverage for several commits now, but the coverage graph oscillates: Similarly, PRs are getting reports of seemingly unrelated drops in coverage. I wonder if the problem might be dates on commits, if we look at the last N commits to the Node.js project:
Note that the ordering of commit dates does not match the order in which they're landed. I'm not 100% sure why our commit history looks like this (CC: @Trott, @nodejs/build). |
That's the date the commit was authored, not merged. That sort of thing will happen on any large codebase with lots of people working on it. For example, here are the last 10 timestamps as of this writing for https://github.com/microsoft/vscode:
If you run |
@Trott thanks for clarifying, and providing the one liner, I figured it was something along these lines. @Trott @thomasrockhu, looking at the list of recent commits on the default branch. It jumps out at me that codecov.io seems to be sorting by |
In #35779, the |
@Qard these are warnings are noisy, but I believe are unrelated (they relate to the file path not being properly calculated for files in the Your coverage output is what I expect |
@bcoe, I'm off this week, but I'll do a deep dive on Monday. My suspicion is that there might be something in the differing number of uploads between the ~87% covered commits (2 uploads) and the ~96% covered commits (1 upload). I would just double-check this article in case anything there is useful. |
@bcoe, looking at the graph here it looks like things have stabilized on the
I have let the product team know about this feedback, thanks so much. I'll get back to you if we have any other questions.
As I mentioned, all of the commits that were still at ~96% had only one build, which looks like a different coverage format than the commits with 2 builds. I think this is a result of PRs not having this commit and the comment you made. Does this make sense?
Is this still an issue for your developers? We have since made fixes here. |
The drop from 96% to 88% was expected; we added C++ coverage, which has a lower percentage than our JavaScript suite, so we saw a drop in coverage when it was merged. The behavior that has been strange, is that it doesn't seem like opened PRs are comparing against the HEAD of our default branch, rather they are comparing against whichever commit has the highest authoring date.
I haven't seen it myself recently, but we're not currently reporting results on a per-PR basis, so there's less incentive for folks to click through to the report. In general, it seems like things have been working well -- I would like to eventually get us reporting coverage information to PRs again, but our nightly reports have seemed healthy. |
Is there an instance here that has been strange? Would love to dig into this and see if there's a bug or a misunderstanding. At the end of the day, if you're happy with the way things are working now, that's what's important. Let me know if there's anything else I can help with here. |
When this issue was opened, Codecov was looking at author date on commits. I believe they are now looking at committer date so I believe this issue is likely solved. Config is updated here: #54019 |
Got an automated coverage report on #35753. The reported coverage regression is estimated as
-8.50%
which is very suspicious, considering that the changes made by the PR are relatively small and tests are included. Impacted files list also includes mostly unrelated files. Is it possible that there is something wrong with codecov configuration or the service itself?See #35753 (comment) and https://codecov.io/gh/nodejs/node/pull/35753 for more details
The text was updated successfully, but these errors were encountered: