[Installer]: configure endpoint/region of registry S3 backing

[mrsimonemms]

Description

There is no way to independently control the bucket configuration for the main S3 bucket and the registry's S3 backing. This introduces the configuration surface in the Installer and implements it in the KOTS installer. The registry persistence is also disabled if the S3 backing is used - this never becomes ready, so causes Helm to timeout.

There is at least one customer blocked by this. If the object storage wasn't configured and the registry S3 was, it also prevented the Helm install job from ever finishing - see internal discussion.

Related Issue(s)

Fixes #9800

How to test

Configure S3 backing for registry and the main object storage using the in-cluster option.

I have checked that there's no regression in the in-cluster repo too

Release Notes

[Installer]: configure endpoint/region of registry S3 backing

Documentation

[mrsimonemms]

/werft no-preview publish-to-kots

👎 unknown command: no-preview
Use /werft help to list the available commands

[mrsimonemms]

/werft run no-preview publish-to-kots

👍 started the job as gitpod-build-sje-registry-s3-fork.0
(with .werft/ from main)

[mrsimonemms]

/werft run no-preview publish-to-kots

👍 started the job as gitpod-build-sje-registry-s3-fork.1
(with .werft/ from main)

[mrsimonemms]

/werft run no-preview publish-to-kots

👍 started the job as gitpod-build-sje-registry-s3-fork.2
(with .werft/ from main)

[mrsimonemms]

This covers some of the same ground as #10208 by @nandajavarma

[nandajavarma]

@mrsimonemms This works well! Just that now that the openSSH install PR is in, this is conflicting. Could you resolve this with a rebase?

[mrsimonemms]

/werft run no-preview publish-to-kots

👍 started the job as gitpod-build-sje-registry-s3-fork.3
(with .werft/ from main)

[nandajavarma]