Build Network Fixtures and Run Tests with Kitchen-Terraform #33

Jberlinsky · 2018-11-20T17:54:34Z

Continuation of #20, incorporating #20 (comment).

Addresses #28 and #25 along the way.

aaron-lane

We should try to limit the use of attributes to values which are necessary to obtain subjects. Relying on an attribute to provide the expected value of an expectations may indicate that the expectation is simply validating that the Terraform resources applied correctly rather than verifying broader intent of the configuration.

We should try to limit expectations to 1 per example in order to provide clarity if expectations fail.

Using the includes and including matchers could help to simplify the examples which rely on filtered values.

let(:taints) { nodes.first.spec.taints.map(&:to_h) }
# ...
expect(taints).to include(
  including(
    effect: "PreferNoSchedule",
    key: "all-pools-example",
    value: "true"
  ),
  # ...
)

Makefile

test/integration/node_pool/controls/gcloud.rb

test/integration/node_pool/controls/kubectl.rb

test/integration/simple_regional/controls/gcloud.rb

test/integration/simple_zonal/controls/gcloud.rb

test/integration/node_pool/controls/gcloud.rb

test/integration/simple_zonal/controls/gcloud.rb

aaron-lane

I have verified that the tests pass. stub_domains requires 2 converges to correctly configure module.gke.google_container_cluster.primary.

morgante · 2018-12-06T19:26:11Z

@Jberlinsky Here are follow-up items:

Converge testing documentation and approach with project factory. We should be pushing people to run tests in Docker to minimize differences.
I think the approach of manually managing fixtures by copying data into arbitrary example directories is very brittle and we need to back out of it ASAP. Principle: I should be able to nuke my local test directory and all testing artifacts will disappear.
I'm still ambivalent about the approach of embedding network setup in the tests. In particular, I'm concerned about how we test shared VPC usage - do we have a plan for that?
kitchen converge fails:

       Error: Error applying plan:
       
       1 error(s) occurred:
       
       * module.gke.google_container_cluster.primary: Resource 'data.google_compute_network.gke_network' not found for variable 'data.google_compute_network.gke_network.self_link'

kitchen destroy fails if converge failed:

       Error: Error applying plan:
       
       1 error(s) occurred:
       
       * module.gke.local.cluster_master_auth_list_layer2: local.cluster_master_auth_list_layer2: At column 40, line 1: list "local.cluster_master_auth_list_layer1" does not have any elements so cannot determine type. in:
       
       ${local.cluster_master_auth_list_layer1[0]}

Jberlinsky · 2018-12-06T19:38:45Z

@morgante I haven't seen the kitchen converge error you mentioned before. Could you expand on how your project was set up, and possibly provide a full log of the test run?

Jberlinsky · 2018-12-13T02:22:12Z

@morgante I've addressed the action items you listed above, based on our conversation on Thursday:

Documentation has been updated to better reflect that of project-factory (which resolves Make testing procedure more clear #12)
The modules in examples/ are encapsulated by test cases in test/fixtures/<example_name>, which creates the requisite fixture network and calls the example as a module. These fixtures are can vary by test case, which will leave ample flexibility to cover shared VPC cases.
The issue you encountered with kitchen converge in a project created by project-factory has been resolved (the solution resolves Allow cluster service account to be overridden without specifying in var.node_pools #23 and Update examples to include example of setting service account #39)

Could you take another look at this when you get a moment?

Jberlinsky · 2018-12-13T02:23:21Z

Worth pointing out that 7e0c063 addresses #23 in passing.

aaron-lane

Just some small changes and clarifications requested.

test/fixtures/deploy_service/example.tf

test/fixtures/deploy_service/outputs.tf

test/make.sh

.ruby-version

README.md

Jberlinsky · 2018-12-13T22:07:50Z

@aaron-lane I've addressed your comments -- would appreciate a re-review!

morgante

Overall this is in much better shape than before. Still a few small warts.

.kitchen.yml

cluster_zonal.tf

examples/node_pool/main.tf

examples/shared_vpc/README.md

test/fixtures/all_examples/install.sh

README.md

cluster_zonal.tf

morgante · 2018-12-19T01:47:45Z

scripts/kubectl_wrapper.sh

@@ -39,7 +39,9 @@ mkdir "${TMPDIR}"

 export KUBECONFIG="${TMPDIR}/config"

-echo "${CA_CERTIFICATE}" | base64 --decode > "${TMPDIR}/ca_certificate"
+# shellcheck disable=SC1117
+base64 --help | grep "\--decode" && B64_ARG="--decode" || B64_ARG="-d"


This seems very complex? Why do we need to do all this greping?

Anyways. This is not a testing change = separate PR please.

This identifies which flag is used to "decode"; the interface of base64 differs across platforms.

I'd argue that this is indeed a testing change -- in moving from running these tests on the local host (the current state of master) to docker, it was necessary to change the interface of base64 as Aaron mentioned above depending on where the test is being run.

scripts/wait-for-cluster.sh

test/fixtures/shared/variables.tf

morgante · 2018-12-19T01:59:35Z

This is very close to being ready, here's my proposed plan of attack.

Blocking issues (cannot merge until fixed):

We can merge but I want us to immediately back out any functional changes into their own PRs. Items to address soon after:

morgante · 2018-12-19T02:03:06Z

Running kitchen converge twice causes a permadiff for networking:

         network:    "projects/clf-cft-testing-dev/global/networks/cft-gke-test-cwky" => "https://www.googleapis.com/compute/v1/projects/clf-cft-testing-dev/global/networks/cft-gke-test-cwky"
         subnetwork: "projects/clf-cft-testing-dev/regions/us-east4/subnetworks/cft-gke-test-cwky" => "https://www.googleapis.com/compute/v1/projects/clf-cft-testing-dev/regions/us-east4/subnetworks/cft-gke-test-cwky"

This is possibly an upstream issue.

test/fixtures/shared/variables.tf

Obsolete

Jberlinsky · 2018-12-19T16:51:20Z

@morgante At a quick glance, the permadiff does indeed look to be upstream. I'm digging into it now.

Jberlinsky · 2018-12-19T17:34:41Z

It does indeed look like the permadiff is upstream -- all the way up to terraform core. See:

google_container_cluster network information keeps being updated on subsequent identical terraform plan hashicorp/terraform-provider-google#1566
Diffs are not getting suppressed when lifecycle.ignore_changes is ignoring a change to a different field hashicorp/terraform#18209

The good news is that as of 15 days ago, it's claimed to be fixed in master.

…ntext with downstream examples

This fixes the following build issue: ``` ERROR: unsatisfiable constraints: go-1.10.5-r0: breaks: world[go=1.10.1-r0] ```

This fixes the following build issue: ``` ERROR: unsatisfiable constraints: curl-7.61.1-r1: breaks: world[curl=7.61.1-r0] git-2.18.1-r0: breaks: world[git=2.18.0-r0] ```

This fixes the following build issue: ``` ERROR: unsatisfiable constraints: ca-certificates-20171114-r3: breaks: world[ca-certificates=20161130-r0] satisfies: curl-7.61.1-r1[ca-certificates] libcurl-7.61.1-r1[ca-certificates] .ruby-rundeps-0[ca-certificates] bash-4.4.19-r1: breaks: world[bash=4.3.42-r5] curl-7.61.1-r1: breaks: world[curl=7.60.0-r1] musl-dev-1.1.19-r10: breaks: world[musl-dev=1.1.14-r16] satisfies: libc-dev-0.7.1-r0[musl-dev] g++-6.4.0-r9: breaks: world[g++=5.3.0-r0] git-2.18.1-r0: breaks: world[git=2.8.6-r0] jq-1.6_rc1-r1: breaks: world[jq=1.5-r2] make-4.2.1-r2: breaks: world[make=4.1-r1] python2-2.7.15-r1: breaks: world[python=2.7.14-r0] satisfies: python2-dev-2.7.15-r1[python2=2.7.15-r1] py-setuptools-39.1.0-r0[python2] py2-pip-10.0.1-r0[python2] py2-pip-10.0.1-r0: breaks: world[py-pip=8.1.2-r0] python2-dev-2.7.15-r1: breaks: world[python-dev=2.7.14-r0] ```

…m under test

… array elements

… to be passed to cluster node pool creation

…g Terraform runs or destroying resources

…modules/feature/automatic-fixture-network-creation Build Network Fixtures and Run Tests with Kitchen-Terraform

Jberlinsky requested a review from adrienthebo November 20, 2018 20:00

morgante requested review from lilithmooncohen and aaron-lane November 26, 2018 18:40

aaron-lane reviewed Nov 28, 2018

View reviewed changes

aaron-lane approved these changes Dec 3, 2018

View reviewed changes

Jberlinsky requested a review from morgante December 13, 2018 02:33

Jberlinsky assigned Jberlinsky and morgante Dec 13, 2018

aaron-lane suggested changes Dec 13, 2018

View reviewed changes

test/fixtures/deploy_service/example.tf Outdated Show resolved Hide resolved

test/fixtures/deploy_service/outputs.tf Show resolved Hide resolved

test/make.sh Show resolved Hide resolved

.ruby-version Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

aaron-lane previously approved these changes Dec 14, 2018

View reviewed changes

Jberlinsky force-pushed the feature/automatic-fixture-network-creation branch from 2bbf53a to 12350cc Compare December 14, 2018 17:20

morgante suggested changes Dec 19, 2018

View reviewed changes

morgante reviewed Dec 19, 2018

View reviewed changes

test/fixtures/shared/variables.tf Outdated Show resolved Hide resolved

Jberlinsky force-pushed the feature/automatic-fixture-network-creation branch from f6ca3a7 to 41a068d Compare December 19, 2018 02:48

Jberlinsky added 6 commits December 19, 2018 13:12

Fix Docker build

1f88f47

Specify cluster versions in examples

45618e7

Set up examples to run in test fixture networks

47189ea

Add network fixtures

98eb6d1

Create fixtures as part of the integration test process, and share co…

5ec503a

…ntext with downstream examples

Clean up botched merge

76b3498

aaron-lane and others added 24 commits December 19, 2018 13:13

Pin Ruby to v2.5 in Makefile

bb0e0dd

Update go to v1.10.5-r0 in Dockerfile

a6045dc

This fixes the following build issue: ``` ERROR: unsatisfiable constraints: go-1.10.5-r0: breaks: world[go=1.10.1-r0] ```

Update curl, git in Dockerfile

a75e252

This fixes the following build issue: ``` ERROR: unsatisfiable constraints: curl-7.61.1-r1: breaks: world[curl=7.61.1-r0] git-2.18.1-r0: breaks: world[git=2.18.0-r0] ```

Refactor example cases with mutliple assertions

486c90d

Reduce attributes passed into test-kitchen examples to simplify syste…

72899fa

…m under test

Switch to using includes matchers instead of filtering for specific…

ae7b5da

… array elements

Enable project-factory support by allowing arbitrary service accounts…

9f8f58e

… to be passed to cluster node pool creation

Call examples as modules from test fixtures

d3d4436

Bump tested GKE versions

9bb3963

Wait for GKE cluster to not be in RECONCILING status before completin…

176ad6f

…g Terraform runs or destroying resources

Fix linter error

0f3fcc1

Pin Ruby to 2.5.1

1c9754c

Update testing documentation

1a84f32

Add line breaks

73afe8b

Bump ruby version to 2.5.3

614c24e

Run terraform fmt as part of linting

be970a4

Fix Go version in tests.

8d46662

Actually symlink terraform.tfvars

b017249

Missed one tfvars symlink

cf5c27a

Add descriptions to test environment variables

4eeaaa3

terraform fmt

a4a3f71

Bring back shared_vpc example and simple test fixture

90a122e

Only .gitignore actual terraform variables, not symlinks

4a32e01

Jberlinsky force-pushed the feature/automatic-fixture-network-creation branch from b16c1d3 to 4a32e01 Compare December 19, 2018 18:13

morgante merged commit 86baa96 into master Dec 19, 2018

morgante mentioned this pull request Dec 27, 2018

Initial import of fetching a secret terraform-google-modules/terraform-google-secret#2

Merged

Jberlinsky mentioned this pull request Jan 3, 2019

Document requirements for running test-kitchen tests #25

Closed

aaron-lane deleted the feature/automatic-fixture-network-creation branch April 10, 2019 16:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build Network Fixtures and Run Tests with Kitchen-Terraform #33

Build Network Fixtures and Run Tests with Kitchen-Terraform #33

Jberlinsky commented Nov 20, 2018 •

edited

Loading

aaron-lane left a comment

aaron-lane left a comment

morgante commented Dec 6, 2018

Jberlinsky commented Dec 6, 2018

Jberlinsky commented Dec 13, 2018 •

edited

Loading

Jberlinsky commented Dec 13, 2018

aaron-lane left a comment

Jberlinsky commented Dec 13, 2018

morgante left a comment

morgante Dec 19, 2018

aaron-lane Dec 19, 2018

Jberlinsky Dec 19, 2018

morgante commented Dec 19, 2018 •

edited

Loading

morgante commented Dec 19, 2018

Jberlinsky commented Dec 19, 2018

Jberlinsky commented Dec 19, 2018 •

edited

Loading

Build Network Fixtures and Run Tests with Kitchen-Terraform #33

Build Network Fixtures and Run Tests with Kitchen-Terraform #33

Conversation

Jberlinsky commented Nov 20, 2018 • edited Loading

aaron-lane left a comment

Choose a reason for hiding this comment

aaron-lane left a comment

Choose a reason for hiding this comment

morgante commented Dec 6, 2018

Jberlinsky commented Dec 6, 2018

Jberlinsky commented Dec 13, 2018 • edited Loading

Jberlinsky commented Dec 13, 2018

aaron-lane left a comment

Choose a reason for hiding this comment

Jberlinsky commented Dec 13, 2018

morgante left a comment

Choose a reason for hiding this comment

morgante Dec 19, 2018

Choose a reason for hiding this comment

aaron-lane Dec 19, 2018

Choose a reason for hiding this comment

Jberlinsky Dec 19, 2018

Choose a reason for hiding this comment

morgante commented Dec 19, 2018 • edited Loading

morgante commented Dec 19, 2018

Jberlinsky commented Dec 19, 2018

Jberlinsky commented Dec 19, 2018 • edited Loading

Jberlinsky commented Nov 20, 2018 •

edited

Loading

Jberlinsky commented Dec 13, 2018 •

edited

Loading

morgante commented Dec 19, 2018 •

edited

Loading

Jberlinsky commented Dec 19, 2018 •

edited

Loading