apigateway/integration: Fix parameters issues

[YakDriver]

Description

This update introduces a two-stage approach to handle AWS request and cache key parameter updates more reliably due to observed challenges in API Gateway parameter behavior:

• Request parameter updates can alter cache key parameters (see [Bug]: Error: updating API Gateway Integration: operation error API Gateway: UpdateIntegration, https response error StatusCode: 400, RequestID: ffd09554-7291-4bbe-963e-359592f90c38, BadRequestException: Invalid mapping expression specified: Validation Result: warnings : [], errors : [Invalid mapping expression parameter specified: method.request.querystring.ucid] #39801).
• Adding one cache key parameter may unexpectedly remove another (see [Bug]: API Gateway cache key parameters unset during apply #29910, Existing cache_key_parameters become uncached when new parameters are added #25736).
• Cache key parameters can be updated independently of request parameters.

Keeping Terraform state in sync with what is in place on AWS is problematic due to these unpredictable, from Terraform's perspective, changes. For example, trying to remove or replace a parameter that state says exists but doesn't actually exist, causes not-found-type errors. Another example is simultaneously performing multiple operations that have the net effect of removing a parameter--the first succeeds but the second gives a not-found-type error.

The sorts of errors seen:

BadRequestException: Invalid mapping expression specified: Validation Result: warnings : [], errors : [Invalid mapping expression parameter specified: method.request.querystring.X-Some-Header-2]

NotFoundException: Invalid parameter name specified

Although not reported, this update also fixes a variety of issues that would have caused problems in some configurations. For example, cache key parameters were not escaped, limiting the naming options without errors.

The new approach:

Stage 1: Request Parameter Updates Only
In the first stage, we update everything except cache key parameters.

Stage 2: Cache Key Parameter Adjustments
After examining results from the first stage (reading AWS rather than relying solely on state), we update the cache key parameters accordingly. For example, replacing (adding back) a poor parameter that got waylaid on her way to visit her grandmother, deep in the forest, with a freshly baked, towering croquembouche. This two-stage approach ensures alignment between the state and AWS.

Relations

Closes #39801
Relates #29991
Closes #29910
Closes #25736

References

The laudable attempt at a fix in #29991 seems to be aimed at the same problem. Unfortunately, that fix was put in the method resource rather than integration. The integration update in the method resource confusingly looked like it was coming from the integration resource itself. In addition, debugging shows that the integration update in the method resource always calls Update with an empty set of operations, occasionally causing errors. We’re keeping the #29991 approach in place to avoid potential edge case disruptions, but it may be safe to remove it in the future. For now, we've gated off the integration update in the method resource so that it only calls Update when there are operations to perform.

Output from Acceptance Testing

% make t T=TestAccAPIGatewayIntegration_ K=apigateway 
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.23.2 test ./internal/service/apigateway/... -v -count 1 -parallel 20 -run='TestAccAPIGatewayIntegration_'  -timeout 360m
2024/11/13 22:47:00 Initializing Terraform AWS Provider...
=== RUN   TestAccAPIGatewayIntegration_basic
=== PAUSE TestAccAPIGatewayIntegration_basic
=== RUN   TestAccAPIGatewayIntegration_contentHandling
=== PAUSE TestAccAPIGatewayIntegration_contentHandling
=== RUN   TestAccAPIGatewayIntegration_Parameters_cacheKey
=== PAUSE TestAccAPIGatewayIntegration_Parameters_cacheKey
=== RUN   TestAccAPIGatewayIntegration_Parameters_cacheKeyUpdate
=== PAUSE TestAccAPIGatewayIntegration_Parameters_cacheKeyUpdate
=== RUN   TestAccAPIGatewayIntegration_Parameters_requestUpdate
=== PAUSE TestAccAPIGatewayIntegration_Parameters_requestUpdate
=== RUN   TestAccAPIGatewayIntegration_Parameters_requestCacheKeyUpdate
=== PAUSE TestAccAPIGatewayIntegration_Parameters_requestCacheKeyUpdate
=== RUN   TestAccAPIGatewayIntegration_integrationType
=== PAUSE TestAccAPIGatewayIntegration_integrationType
=== RUN   TestAccAPIGatewayIntegration_TLS_insecureSkipVerification
=== PAUSE TestAccAPIGatewayIntegration_TLS_insecureSkipVerification
=== RUN   TestAccAPIGatewayIntegration_disappears
=== PAUSE TestAccAPIGatewayIntegration_disappears
=== CONT  TestAccAPIGatewayIntegration_basic
=== CONT  TestAccAPIGatewayIntegration_Parameters_requestCacheKeyUpdate
=== CONT  TestAccAPIGatewayIntegration_integrationType
=== CONT  TestAccAPIGatewayIntegration_TLS_insecureSkipVerification
=== CONT  TestAccAPIGatewayIntegration_disappears
=== CONT  TestAccAPIGatewayIntegration_Parameters_cacheKeyUpdate
=== CONT  TestAccAPIGatewayIntegration_Parameters_requestUpdate
=== CONT  TestAccAPIGatewayIntegration_Parameters_cacheKey
=== CONT  TestAccAPIGatewayIntegration_contentHandling
--- PASS: TestAccAPIGatewayIntegration_contentHandling (42.27s)
--- PASS: TestAccAPIGatewayIntegration_basic (75.44s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_requestCacheKeyUpdate (108.11s)
--- PASS: TestAccAPIGatewayIntegration_disappears (204.04s)
--- PASS: TestAccAPIGatewayIntegration_TLS_insecureSkipVerification (206.17s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_cacheKeyUpdate (268.23s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_cacheKey (374.39s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_requestUpdate (423.50s)
--- PASS: TestAccAPIGatewayIntegration_integrationType (672.06s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/apigateway	675.985s
% make t T=TestAccAPIGatewayMethod_ K=apigateway
make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.23.2 test ./internal/service/apigateway/... -v -count 1 -parallel 20 -run='TestAccAPIGatewayMethod_'  -timeout 360m
2024/11/13 22:57:27 Initializing Terraform AWS Provider...
=== RUN   TestAccAPIGatewayMethod_basic
=== PAUSE TestAccAPIGatewayMethod_basic
=== RUN   TestAccAPIGatewayMethod_customAuthorizer
=== PAUSE TestAccAPIGatewayMethod_customAuthorizer
=== RUN   TestAccAPIGatewayMethod_cognitoAuthorizer
=== PAUSE TestAccAPIGatewayMethod_cognitoAuthorizer
=== RUN   TestAccAPIGatewayMethod_customRequestValidator
=== PAUSE TestAccAPIGatewayMethod_customRequestValidator
=== RUN   TestAccAPIGatewayMethod_disappears
=== PAUSE TestAccAPIGatewayMethod_disappears
=== RUN   TestAccAPIGatewayMethod_operationName
=== PAUSE TestAccAPIGatewayMethod_operationName
=== CONT  TestAccAPIGatewayMethod_basic
=== CONT  TestAccAPIGatewayMethod_customRequestValidator
=== CONT  TestAccAPIGatewayMethod_disappears
=== CONT  TestAccAPIGatewayMethod_cognitoAuthorizer
=== CONT  TestAccAPIGatewayMethod_operationName
=== CONT  TestAccAPIGatewayMethod_customAuthorizer
--- PASS: TestAccAPIGatewayMethod_basic (23.36s)
--- PASS: TestAccAPIGatewayMethod_customRequestValidator (23.38s)
--- PASS: TestAccAPIGatewayMethod_customAuthorizer (92.07s)
--- PASS: TestAccAPIGatewayMethod_operationName (180.28s)
--- PASS: TestAccAPIGatewayMethod_cognitoAuthorizer (180.39s)
--- PASS: TestAccAPIGatewayMethod_disappears (191.39s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/apigateway	195.316s

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[johnsonaj]

LGTM 🚀

% make testacc TESTARGS='-run=TestAccAPIGatewayIntegration_' PKG=apigateway

make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.23.2 test ./internal/service/apigateway/... -v -count 1 -parallel 20  -run=TestAccAPIGatewayIntegration_ -timeout 360m
2024/11/14 10:50:48 Initializing Terraform AWS Provider...
--- PASS: TestAccAPIGatewayIntegration_disappears (17.87s)
--- PASS: TestAccAPIGatewayIntegration_contentHandling (49.39s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_requestCacheKeyUpdate (80.20s)
--- PASS: TestAccAPIGatewayIntegration_TLS_insecureSkipVerification (122.72s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_requestUpdate (134.10s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_cacheKey (238.85s)
--- PASS: TestAccAPIGatewayIntegration_Parameters_cacheKeyUpdate (251.98s)
--- PASS: TestAccAPIGatewayIntegration_basic (413.99s)
--- PASS: TestAccAPIGatewayIntegration_integrationType (654.65s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/apigateway	661.234s

% make testacc TESTARGS='-run=TestAccAPIGatewayMethod_' PKG=apigateway

make: Verifying source code with gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go1.23.2 test ./internal/service/apigateway/... -v -count 1 -parallel 20  -run=TestAccAPIGatewayMethod_ -timeout 360m
2024/11/14 11:03:34 Initializing Terraform AWS Provider...
--- PASS: TestAccAPIGatewayMethod_disappears (15.48s)
--- PASS: TestAccAPIGatewayMethod_customAuthorizer (46.44s)
--- PASS: TestAccAPIGatewayMethod_basic (79.41s)
--- PASS: TestAccAPIGatewayMethod_customRequestValidator (114.61s)
--- PASS: TestAccAPIGatewayMethod_operationName (226.70s)
--- PASS: TestAccAPIGatewayMethod_cognitoAuthorizer (444.29s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/apigateway	450.735s

[github-actions]

This functionality has been released in v5.76.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.