Kubernetes Resource Naming Convention Needs Rethinking

[gkramer]

Attempting to spin up two separate k8s clusters within the same stack, both of which run generic services (external-dns, cert-manager, etc) fails due to URN naming convention. It seems that URN should include the cluster name, so as to facilitate this requirement. This problem becomes more pronounced when deploying CRDs, where its not possible to modify the resource/CRD name.

Steps to reproduce

Deploy two K8S clusters within the same stack, both of which are running cert-manager. Pulumi will fail due to lack of unique URNs.

[mikhailshilkov]

URNs are defined by the name of the Pulumi resource which doesn't have to match the actual resource name in the provider's target. If you have two K8s clusters in the same stack, you need to give them unique Pulumi names. The same applies to other resource types. Component resources will usually use the parent name to build up the children names.

Do you want to share some code where you think it's problematic?

[travisjeffery]

The problem is deploying the resources. Pulumi doesn't use the provider name or k8s name in the URN of the resources. So if you deploy the same resources to two different k8s clusters, Pulumi generates the same URN.

As an example, try using NewConfigFile with this CSI AWS provider. And then doing:

_, err = k8syaml.NewConfigFile(ctx, clusterName+"_aws_secrets_store_provider", &k8syaml.ConfigFileArgs{
		File: "aws-provider-installer.yaml",
}, provider)
if err != nil {
	return err
}

On two different clusterNames and providers configured for two different k8s clusters. Pulumi still thinks it's managing the same resources meanwhile they're on different k8s clusters.

[travisjeffery]

I ended up just writing everything in that yaml in the equivalent resource code by hand, but that api is pretty broken. Should at least use the logical name given to help make things unique.

[TheDome]

I have the same problem using the Pulumi TypeScript SDK and Azure-native Kubernetes 👍

[EronWright]

The solution is to use the resourcePrefix property to make the child resource names be unique. For example:

_, err = k8syaml.NewConfigFile(ctx, "cluster1", &k8syaml.ConfigFileArgs{
	File: "aws-provider-installer.yaml",
	ResourcePrefix: "cluster1",
}, provider)

_, err = k8syaml.NewConfigFile(ctx, "cluster2", &k8syaml.ConfigFileArgs{
	File: "aws-provider-installer.yaml",
	ResourcePrefix: "cluster2",
}, provider)

Note that the "v2" version of the ConfigGroup resource automatically uses its own name as a prefix for the child resource names. Also, the prefix doesn't affect the Kubernetes object name.