Run Windows tests by default

[pjanotti]

Component(s)

No response

Describe the issue you're reporting

As of August of 2024 Windows tests are run only when the label Run Windows is added to a PR. The result is that many PRs, that in principle are not specific to Windows, end up being merged without running tests on Windows. Due to subtle differences between the OSes (scheduling, file systems, etc) we end up with bugs and flaky Windows tests. These are only caught in unrelated PRs (that have the label) or after merge when Windows tests are ran on main.

This issue is to track the eventual deprecation of the Run Windows label, i.e.: in the end all tests should run by default on Windows.

I will keep updating this top issue if identified issues and corresponding fixes. This is expected to take some time.

[crobert-1]

I'm concerned about the potential impact of running Windows tests on every PR. I know as it is the contrib repo puts a pretty heavy burden on CNCF's GitHub runners, and this would just add to it.

I don't have a ton of context here though, others would likely have more informed input. It's not a blocker for me, just wanted to call it out.

[pjanotti]

That's a good point @crobert-1 - in general it would be better if we just ran the tests of the affected components, that said this seems a separate issue that can be handled concurrently to this one and even have it as a pre-requisite to finally enabling Windows tests on every PR.

[pjanotti]

I will keep monitoring the runs, but, it seems that they are in much better shape right now.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[mx-psi]

From the Collector SIG meeting on 2024-12-11: a net improvement would be to automatically add "Run Windows" on PRs that modify certain files/components that we know run on Windows. This would not be perfect but it would improve coverage and prevent some issues from happening on main.