Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

d/aws_s3_bucket_object: Don't crash when S3 HeadObject returns empty response #14154

Merged
merged 7 commits into from
Oct 6, 2020
Merged

d/aws_s3_bucket_object: Don't crash when S3 HeadObject returns empty response #14154

merged 7 commits into from
Oct 6, 2020

Conversation

ewbankkit
Copy link
Contributor

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #14151.
Relates #12992.
Relates #7584.

Release note for CHANGELOG:

data-source/aws_s3_bucket_object: Don't crash when S3 `HeadObject` API returns empty response.

Output from acceptance testing:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_' 
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_ -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_basic
=== PAUSE TestAccDataSourceAWSS3BucketObject_basic
=== RUN   TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint
=== PAUSE TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint
=== RUN   TestAccDataSourceAWSS3BucketObject_readableBody
=== PAUSE TestAccDataSourceAWSS3BucketObject_readableBody
=== RUN   TestAccDataSourceAWSS3BucketObject_kmsEncrypted
=== PAUSE TestAccDataSourceAWSS3BucketObject_kmsEncrypted
=== RUN   TestAccDataSourceAWSS3BucketObject_allParams
=== PAUSE TestAccDataSourceAWSS3BucketObject_allParams
=== RUN   TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff
=== PAUSE TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff
=== RUN   TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn
=== PAUSE TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn
=== RUN   TestAccDataSourceAWSS3BucketObject_LeadingSlash
=== PAUSE TestAccDataSourceAWSS3BucketObject_LeadingSlash
=== RUN   TestAccDataSourceAWSS3BucketObject_MultipleSlashes
=== PAUSE TestAccDataSourceAWSS3BucketObject_MultipleSlashes
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_basic
=== CONT  TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff
=== CONT  TestAccDataSourceAWSS3BucketObject_MultipleSlashes
=== CONT  TestAccDataSourceAWSS3BucketObject_allParams
=== CONT  TestAccDataSourceAWSS3BucketObject_LeadingSlash
=== CONT  TestAccDataSourceAWSS3BucketObject_kmsEncrypted
=== CONT  TestAccDataSourceAWSS3BucketObject_readableBody
=== CONT  TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (42.12s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basic (45.22s)
--- PASS: TestAccDataSourceAWSS3BucketObject_readableBody (47.39s)
--- PASS: TestAccDataSourceAWSS3BucketObject_allParams (47.41s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff (48.20s)
--- PASS: TestAccDataSourceAWSS3BucketObject_kmsEncrypted (48.49s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint (48.81s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn (49.41s)
--- PASS: TestAccDataSourceAWSS3BucketObject_LeadingSlash (67.87s)
--- PASS: TestAccDataSourceAWSS3BucketObject_MultipleSlashes (68.27s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	68.374s

@ewbankkit ewbankkit requested a review from a team July 12, 2020 22:38
@ghost ghost added size/S Managed by automation to categorize the size of a PR. provider Pertains to the provider itself, rather than any interaction with AWS. service/s3 Issues and PRs that pertain to the s3 service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Jul 12, 2020
@ewbankkit ewbankkit added bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. and removed provider Pertains to the provider itself, rather than any interaction with AWS. labels Jul 12, 2020
@DrFaust92 DrFaust92 requested review from DrFaust92 and removed request for a team July 31, 2020 21:09
DrFaust92
DrFaust92 approved these changes Aug 24, 2020
View reviewed changes
Copy link
Collaborator

@DrFaust92 DrFaust92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ewbankkit
d/aws_s3_bucket_object: Add 'TestAccDataSourceAWSS3BucketObject_Singl…
8fc9d9a 
…eSlashAsKey'.

Acceptance test output:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x3d83617]

goroutine 867 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.dataSourceAwsS3BucketObjectRead(0xc00040b9d0, 0x5e9e540, 0xc0003fca00, 0xc00040b9d0, 0x0)
	/home/kit/wrk/src/github.com/terraform-providers/terraform-provider-aws/aws/data_source_aws_s3_bucket_object.go:168 +0x887
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).ReadDataApply(0xc0001478c0, 0xc000effb80, 0x5e9e540, 0xc0003fca00, 0xc00003b920, 0x1, 0x0)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/resource.go:403 +0x88
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).ReadDataApply(0xc00012a200, 0xc000cdba88, 0xc000effb80, 0xc000effb80, 0x0, 0x0)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/provider.go:451 +0x8f
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadDataSource(0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc00162f500, 0xc0017fd758, 0xc0018d4960, 0xc00008cb30)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/internal/helper/plugin/grpc_provider.go:1036 +0x44e
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadDataSource_Handler(0x68e0540, 0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc002672240, 0x0, 0x796f360, 0xc0018d4960, 0xc0005b2e00, 0x1ab)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/internal/tfplugin5/tfplugin5.pb.go:3341 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0xc0020be1e0, 0xb6896f0, 0x0, 0x0, 0x0)
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:1024 +0x4f4
google.golang.org/grpc.(*Server).handleStream(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0x0)
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:1313 +0xd97
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001f8ee70, 0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00)
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xa1
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	24.265s
FAIL
GNUmakefile:26: recipe for target 'testacc' failed
make: *** [testacc] Error 1
@ewbankkit
Handle 'NoSuchTagSet' errors when listing S3 bucket object tags.
298f82d 
Failing acceptance test:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
--- FAIL: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (24.58s)
    testing.go:684: Step 0 error: errors during apply:

        Error: error listing tags for S3 Bucket (tf-acc-test-2642544354113174819) Object (/): NoSuchTagSet: The TagSet does not exist
        	status code: 404, request id: 3ADF2BE8D6DAD57E, host id: zIz72pHcPivqgl0t79J/9kF02E5N7ibZy3Rof+ggH2+Y3sffOQsjRQVkP5B/PinBmLJnI1nfvCk=

          on /tmp/tf-test734233549/main.tf line 6:
          (source code not available)

    testing.go:745: Error destroying resource! WARNING: Dangling resources
        may exist. The full state and error is shown below.

        Error: errors during refresh: error listing tags for S3 Bucket (tf-acc-test-2642544354113174819) Object (/): NoSuchTagSet: The TagSet does not exist
        	status code: 404, request id: 9746D2A0387F2F59, host id: tCpkS+zPf8miGLqN8BaDPGbPBs03PZoC3iJHWvhd3SeN7k/Y1BD1AKJmujxgfBqGshau1OP9Pu4=

        State: <nil>
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	24.654s
FAIL
GNUmakefile:26: recipe for target 'testacc' failed
make: *** [testacc] Error 1
@ewbankkit
d/aws_s3_bucket_object: Don't crash when HeadObject returns empty res…
833b60d 
…ponse.

Acceptance test output:
$ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (40.69s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	40.749s
@ghost ghost added the provider Pertains to the provider itself, rather than any interaction with AWS. label Oct 6, 2020
@bflad bflad self-assigned this Oct 6, 2020
@ghost ghost added size/XXL Managed by automation to categorize the size of a PR. and removed size/S Managed by automation to categorize the size of a PR. labels Oct 6, 2020
ewbankkit and others added 3 commits October 6, 2020 11:19
@ewbankkit
d/aws_s3_bucket_object: Add 'TestAccDataSourceAWSS3BucketObject_Singl…
6f0a858 
…eSlashAsKey'.

Acceptance test output:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x3d83617]

goroutine 867 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.dataSourceAwsS3BucketObjectRead(0xc00040b9d0, 0x5e9e540, 0xc0003fca00, 0xc00040b9d0, 0x0)
	/home/kit/wrk/src/github.com/terraform-providers/terraform-provider-aws/aws/data_source_aws_s3_bucket_object.go:168 +0x887
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).ReadDataApply(0xc0001478c0, 0xc000effb80, 0x5e9e540, 0xc0003fca00, 0xc00003b920, 0x1, 0x0)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/resource.go:403 +0x88
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).ReadDataApply(0xc00012a200, 0xc000cdba88, 0xc000effb80, 0xc000effb80, 0x0, 0x0)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/provider.go:451 +0x8f
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadDataSource(0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc00162f500, 0xc0017fd758, 0xc0018d4960, 0xc00008cb30)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/internal/helper/plugin/grpc_provider.go:1036 +0x44e
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadDataSource_Handler(0x68e0540, 0xc0017fd758, 0x796f360, 0xc0018d4960, 0xc002672240, 0x0, 0x796f360, 0xc0018d4960, 0xc0005b2e00, 0x1ab)
	/home/kit/wrk/pkg/mod/github.com/hashicorp/[email protected]/internal/tfplugin5/tfplugin5.pb.go:3341 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0xc0020be1e0, 0xb6896f0, 0x0, 0x0, 0x0)
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:1024 +0x4f4
google.golang.org/grpc.(*Server).handleStream(0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00, 0x0)
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:1313 +0xd97
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001f8ee70, 0xc0020ae780, 0x799ef80, 0xc0020ae900, 0xc000f63c00)
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/kit/wrk/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xa1
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	24.265s
FAIL
GNUmakefile:26: recipe for target 'testacc' failed
make: *** [testacc] Error 1
@ewbankkit
d/aws_s3_bucket_object: Don't crash when HeadObject returns empty res…
77ace09 
…ponse.

Acceptance test output:
$ make testacc TEST=./aws/ TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (40.69s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	40.749s
@ewbankkit @bflad
Update aws/internal/keyvaluetags/s3_tags.go
7123ffd 
Co-authored-by: Brian Flad <[email protected]>
@ewbankkit ewbankkit force-pushed the b-aws_s3_bucket_object-data-source-crash-on-nil-etag branch from c9bbf98 to 7123ffd Compare October 6, 2020 15:21
@ghost ghost added size/S Managed by automation to categorize the size of a PR. and removed size/XXL Managed by automation to categorize the size of a PR. labels Oct 6, 2020
@ewbankkit
Copy link
Contributor Author

Not sure why the PR blew up in size there but I have rebased and now it's back down to size/S.
Re-ran acceptance tests:

$ make testacc TEST=./aws TESTARGS='-run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey -timeout 120m
=== RUN   TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== PAUSE TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
=== CONT  TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (18.81s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	18.888s

@bflad bflad added this to the v3.10.0 milestone Oct 6, 2020
bflad
bflad approved these changes Oct 6, 2020
View reviewed changes
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 🚀

Output from acceptance testing:

--- PASS: TestAccDataSourceAWSS3BucketObject_kmsEncrypted (53.21s)
--- PASS: TestAccDataSourceAWSS3BucketObject_readableBody (53.78s)
--- PASS: TestAccDataSourceAWSS3BucketObject_SingleSlashAsKey (54.16s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basic (55.33s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOff (55.08s)
--- PASS: TestAccDataSourceAWSS3BucketObject_ObjectLockLegalHoldOn (56.63s)
--- PASS: TestAccDataSourceAWSS3BucketObject_allParams (57.27s)
--- PASS: TestAccDataSourceAWSS3BucketObject_basicViaAccessPoint (59.73s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_startAfter (74.51s)
--- PASS: TestAccDataSourceAWSS3BucketObject_MultipleSlashes (76.46s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_fetchOwner (76.46s)
--- PASS: TestAccDataSourceAWSS3BucketObject_LeadingSlash (78.07s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_maxKeys (78.09s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_basicViaAccessPoint (77.91s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_basic (78.31s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_all (79.05s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_prefixes (78.72s)
--- PASS: TestAccDataSourceAWSS3BucketObjects_encoded (78.60s)

@bflad bflad merged commit a6a0a96 into hashicorp:master Oct 6, 2020
bflad added a commit that referenced this pull request Oct 6, 2020
@bflad
Update CHANGELOG for #14154
85d8cda
@ewbankkit ewbankkit deleted the b-aws_s3_bucket_object-data-source-crash-on-nil-etag branch October 6, 2020 18:48
@ghost
Copy link

ghost commented Oct 9, 2020

This has been released in version 3.10.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Nov 6, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked as resolved and limited conversation to collaborators Nov 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@DrFaust92 DrFaust92 DrFaust92 approved these changes

@bflad bflad bflad approved these changes

Assignees

@bflad bflad

Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. provider Pertains to the provider itself, rather than any interaction with AWS. service/s3 Issues and PRs that pertain to the s3 service. size/S Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v3.10.0
Development

Successfully merging this pull request may close these issues.

Bad key value in s3 data source causes crash
3 participants
@ewbankkit @bflad @DrFaust92