Consolidate integration tests

[blampe]

This moves integration tests currently under provider into the examples directory, and removes provider_test defined in extraTests.

These integration tests will now benefit from pulumi/ci-mgmt#1033 as a result, and #4911 is no longer necessary.

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

Maintainer note: consult the runbook for dealing with any breaking changes.

[flostadler]

Generally I find it a bit odd that we put the tests (i.e. go files) into the examples/ directory. Couldn't we do the same thing we did for EKS and put all the testdata and programs into provider and reference the example programs from there?

Not blocking on this though. This change is a great improvement!

[blampe]

@flostadler good questions!

Generally I find it a bit odd that we put the tests (i.e. go files) into the examples/ directory.

Yeah, examples has always been a very weird directory.

Couldn't we do the same thing we did for EKS and put all the testdata and programs into provider and reference the example programs from there?

It doesn't matter where the integration tests live -- we currently hard-code it to examples but that will be configurable after pulumi/ci-mgmt#1217. So if you want integration tests to live under provider then go for it.

However I would add that for simplicity it's a good idea to keep slower-running integration tests separate from unit tests. So for example if unit tests live under provider you can run go test ./provider/... without needing to remember -short or build tags or whatnot. In many cases our providers have zero unit tests, which is probably why we've started putting e2e tests with the provider -- no unit tests to commingle with. I think that's a bad pattern, but again it's up to you as the maintainer. At the end of the day we will run "fast" tests on one CI runner, and "slow" tests across several CI runners, and how you make that distinction will be up to you.

I should mention another (self-inflicted) reason for putting integration tests under provider is that it puts them in the same module as the rest of your code, which is a side-effect of us creating separate modules under examples/go.mod and provider/go.mod. This is an anti-pattern, and if the modules are consolidated under one go.mod at the repo root then it really doesn't matter where the tests live because they'll always belong to the same module.

Finally I'll also add that after we stop sharding by language it will no longer be necessary to need build tags on your integration tests, which gives you a much nicer developer experience when you need to touch them.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (e1ae150) to head (9c25997).
Report is 1 commits behind head on master.

Additional details and impacted files

@@      Coverage Diff       @@
##   master   #4910   +/-   ##
==============================
==============================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pulumi-bot]

This PR has been shipped in release v6.65.0.