[Bug]: "aws_db_instance", "storage_type" (io1 > gp3) can't be changed if "allocated_storage" is under 400 GB

[crimean-celica]

Relates #28847
Relates #33512
Relates #33529
Relates #36648
Relates #37257

Terraform Core Version

1.0.9

AWS Provider Version

4.48.0

Affected Resource(s)

aws_db_instance

Expected Behavior

"storage_type" can be changed from "io1" to "gp3"

Actual Behavior

"storage_type" can't be changed (io1 > gp3) as far as you have under 400 GB disk

if you do not specify "iops":

Error: updating RDS DB Instance (delme2): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: 20ed3bd9-1794-4451-a314-6eff87d6e851, api error InvalidParameterCombination: You must specify both the storage size and iops when modifying the storage size or iops on a DB instance that has iops.

if you do specify "iops":

Error: updating RDS DB Instance (inapp-stage-delme2): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: 44cad8d2-dea0-4c77-9979-2314cc8d528a, api error InvalidParameterCombination: You can't specify IOPS or storage throughput for engine postgres and a storage size less than 400.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_db_instance" "delme2" {
  identifier = "delme2"

  engine            = "postgres"
  engine_version    = "13.7"
  instance_class    = "db.t4g.small"
  allocated_storage = 100
  storage_encrypted = true
  apply_immediately = true
  storage_type      = "io1"
  iops              = 1000

  iam_database_authentication_enabled = true

  username = "delme_db_admin"
  password = "P@ssw0rd"
  port     = "5432"

  parameter_group_name = "default.postgres12"

  deletion_protection = false
}

Steps to Reproduce

1. create a new RDS instance ( storage_type = "io1", allocated_storage = 100, iops = 1000 )
2. change storage_type to "gp3" with or without specifying "iops" value

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[crimean-celica]

Technically, it's possible to change io1 > gp2 > gp3 (it works, I tested) but this way will have gp2 (with lower iops) during ~6 hours (storage optimization time).

[fatmcgav-depop]

I was able to reproduce this today when moving a 5TB RDS instance from io1 to gp3.

2023-01-10T10:54:54.771Z [INFO]  provider.terraform-provider-aws_v4.46.0: [INFO] Only settings updating, instance changes will be applied in next maintenance window
2023-01-10T10:54:54.771Z [DEBUG] provider.terraform-provider-aws_v4.46.0: [DEBUG] Waiting for state to become: [success]
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: [DEBUG] [aws-sdk-go-v2] Request
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: POST / HTTP/1.1
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Host: rds.us-east-1.amazonaws.com
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.2.7 (+https://www.terraform.io) terraform-provider-aws/4.46.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.17.2 os/macos lang/go/1.19.3 md/GOOS/darwin md/GOARCH/amd64 api/rds/1.32.0
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Content-Length: 160
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Amz-Sdk-Invocation-Id: 5e2b277b-4365-41c0-8cf9-addcf1e79007
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Amz-Sdk-Request: attempt=1; max=25
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Authorization: AWS4-HMAC-SHA256 Credential=ASIA47JNX4DM3MJWQSEB/20230110/us-east-1/rds/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=**redacted**
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Content-Type: application/x-www-form-urlencoded
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: X-Amz-Date: 20230110T105454Z
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: X-Amz-Security-Token: **redacted**
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Accept-Encoding: gzip
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0:
2023-01-10T10:54:54.772Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Action=ModifyDBInstance&DBInstanceIdentifier=**redacted**&DeletionProtection=true&StorageThroughput=2000&StorageType=gp3&Version=2014-10-31
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: [DEBUG] [aws-sdk-go-v2] Response
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: HTTP/1.1 400 Bad Request
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Connection: close
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Content-Length: 377
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Content-Type: text/xml
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Date: Tue, 10 Jan 2023 10:54:55 GMT
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: Strict-Transport-Security: max-age=31536000
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: X-Amzn-Requestid: 646d7ca3-764f-47bb-be88-9d171bc06714
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0:
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: <ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0:   <Error>
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0:     <Type>Sender</Type>
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0:     <Code>InvalidParameterCombination</Code>
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0:     <Message>You must specify both the storage size and iops when modifying the storage size or iops on a DB instance that has iops.</Message>
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0:   </Error>
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0:   <RequestId>646d7ca3-764f-47bb-be88-9d171bc06714</RequestId>
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: </ErrorResponse>
2023-01-10T10:54:55.214Z [DEBUG] provider.terraform-provider-aws_v4.46.0: [DEBUG] [aws-sdk-go-v2] request failed with unretryable error https response error StatusCode: 400, RequestID: 646d7ca3-764f-47bb-be88-9d171bc06714, api error InvalidParameterCombination: You must specify both the storage size and iops when modifying the storage size or iops on a DB instance that has iops.

I think this boils down to a logic flaw here:

terraform-provider-aws/internal/service/rds/instance.go

Lines 2065 to 2072 in 1076f59

	if d.HasChange("storage_type") {
	needsModify = true
	input.StorageType = aws.String(d.Get("storage_type").(string))
	if aws.StringValue(input.StorageType) == storageTypeIO1 {
	input.Iops = aws.Int32(int32(d.Get("iops").(int)))
	}
	}

[ipolishchuk]

I have the same problem with
terraform 1.3.6
provider hashicorp/aws v4.49.0

[ipolishchuk]

╷
│ Error: updating RDS DB Instance (XXXXXXXXX): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: 4cb24468-753e-4397-8364-6fdd7cad2977, api error InvalidParameterCombination: You must specify both the storage size and iops when modifying the storage size or iops on a DB instance that has iops.
│
│ with module.db_instance-0.module.db_instance.module.db_instance.aws_db_instance.this[0],
│ on ../../../../../modules/terraform-aws-rds/5.2.2/modules/db_instance/main.tf line 32, in resource "aws_db_instance" "this":
│ 32: resource "aws_db_instance" "this" {
│

[fatmcgav]

I've opened #28847 with a change that I think should solve this issue.

Unfortunately I'm unable to run the acceptance tests atm, but will see if I can work out a way to get them run soon...

[sindhu-selvaraj-starling]

This is an issue with AWS CLI command, I am contacting their support now

aws rds modify-db-instance --db-instance-identifier database-1 --storage-type gp3 --apply-immediately                   

An error occurred (InvalidParameterCombination) when calling the ModifyDBInstance operation: You must specify both the storage size and iops when modifying the storage size or iops on a DB instance that has iops.

[sindhu-selvaraj-starling]

When allocated_storage is specified, it allows to do the change via CLI

aws rds modify-db-instance --db-instance-identifier  database-2 --storage-type gp3 --allocated-storage 100 --apply-immediately

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.78.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.