Use KinD clusters in integration tests for presubmits

[rquitales]

Proposed changes

This PR switches the underlying test infrastructure to provision KinD clusters for PR checks. This will drastically speed up the feedback loop from the current ~45 mins to 25 mins. Furthermore, this also pushes the cluster creation step into the test job to enable re-running only the failed job should a test flake occur.

Changes made:

• Updated the GHA workflow to run PR checks in Kind clusters to reduce feedback loop
  • Note: post-submit checks still run against a GKE cluster as some test cases can't don't pass in KinD
• Updated existing tests to either work with Kind if they are simple, or to be skipped if the test needs a full cloud k8s cluster
• Broke apart the Golang tests to run concurrently as GH Action workers now do not OOM
• Fixed some hard-coded test logic that breaks on newer k8s versions

Related issues (optional)

Fixes: #2243

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 26.84%. Comparing base (c59ddff) to head (6f8196d).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2831   +/-   ##
=======================================
  Coverage   26.84%   26.84%           
=======================================
  Files          53       53           
  Lines        7722     7722           
=======================================
  Hits         2073     2073           
  Misses       5476     5476           
  Partials      173      173           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EronWright]

Some questions:

1. Why is it better to have the test code create the cluster instead of a CI step?
2. How/when will we support the new provider test framework and how many tests are being skipped?

[EronWright]

Nit: Seems like we are exploiting the --short flag, not to "tell long-running tests to shorten their run time" but to filter incompatible tests.
Ideally we'd update the tests to be compatible, e.g. by using a clusterIP.

My main feedback is that I would prefer we keep cluster creation outside of the go test code, e.g. by using a GH action in the step immediately preceding test execution.

[rquitales]

Some questions:

1. Why is it better to have the test code create the cluster instead of a CI step?
2. How/when will we support the new provider test framework and how many tests are being skipped?

1. Having a Go test framework handle cluster creation ensures test portability and allows for a unified entrypoint. This removes the requirement for the developer to pre-configure their test environment to target different test environments. It'd be nice if I could simply just run go test --use-gke or go test --use-kind for example without needing to worry if I had set up these environments manually correctly beforehand.
2. The new provider test framework was supported in the last commit.

After offline discussions, I've decided to opt for cluster creation in the CI job instead for now and parking this approach. We might revisit this in the future.

[rquitales]

Nit: Seems like we are exploiting the --short flag, not to "tell long-running tests to shorten their run time" but to filter incompatible tests. Ideally we'd update the tests to be compatible, e.g. by using a clusterIP.

My main feedback is that I would prefer we keep cluster creation outside of the go test code, e.g. by using a GH action in the step immediately preceding test execution.

Fwiw, most of the tests that are skipped due to not being passable in KinD in their current state are also long running tests.

[rquitales]

Note: changes in this file will need to be replicated in ci-mgmt. It is included here for faster feedback loop.

[EronWright]

LGTM!