Fedora 34 cluster

[victorusu]

I have a problem when I try to use the Fedora 34 cloud image from https://alt.fedoraproject.org/cloud/ and the OpenStack version of Magic Castle.
The error is the following:

╷
│ Error: file provisioner error
│ 
│   with module.openstack.module.cluster_config.null_resource.deploy_hieradata[0],
│   on common/cluster_config/main.tf line 74, in resource "null_resource" "deploy_hieradata":
│   74:   provisioner "file" {
│ 
│ timeout - last error: Error connecting to bastion: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
╵
╷
│ Error: file provisioner error
│ 
│   with module.openstack.module.instance_config.null_resource.deploy_hieradata[0],
│   on common/instance_config/main.tf line 41, in resource "null_resource" "deploy_hieradata":
│   41:   provisioner "file" {
│ 
│ timeout - last error: Error connecting to bastion: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
╵

This error doesn’t happen when using cloud images of RockyLinux, CentOS8.4, and CentOS-Stream.

Does anyone know how to fix this?

I tried the solutions described here, where I tried to the private key to the ssh-agent

$ ssh-add </path/to/private/key>

and here, where I tried to add a depends_on.

...
 resource "null_resource" "deploy_hieradata" {                                      
   count = contains(local.all_tags, "puppet") && contains(local.all_tags, "public") ? 1 : 0 
+  depends_on = [var.instances]                                                     
                                                                                   
   connection {  
...

Since the issue is not seen on RockyLinux, CentOS8.4, and CentOS-Stream, I wonder if there is anything special or different in the fedora 34 cloud image. Perhaps something that I need to enable?

Thanks in advance.

[cmd-ntrf]

I had to do some digging to find an answer because it is was not clear at first what was happening.

Magic Castle uploads some files using Terraform file provisionner in two modules: cluster_config and dns. To upload these files, Terraform relies on golang x/crypto/ssh library to create an ssh connection to the cluster.

The code to upload a file to the cluster can simplified to the following snippet:

resource "null_resource" "deploy_file" {

  connection {
    type        = "ssh"
    host        = "cluster.domain.tld"
    user        = "centos"
  }

  provisioner "file" {
    content     = "---"
    destination = "data.yaml"
  }
}

When I tried the preceding snippet with a barebone Fedora34 instance, Terraform was unable to upload the file and returned:

ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

which lead me to dig into Terraform file provisioner issue, and I found this:
hashicorp/terraform#27768
which concludes that golang x/crypto/ssh library is to blame and there is not much we can do for now.

[victorusu]

I managed to make it work using ECDSA keys, instead of RSA ones.
The hack that I used was to change the common/instance_config/main.tf file to the following:

#resource "tls_private_key" "ssh" {                                                
#  count     = var.generate_ssh_key ? 1 : 0                                        
#  algorithm = "RSA"                                                               
#  rsa_bits  = 4096                                                                
#}                                                                                 
#                                                                                  
#resource "tls_private_key" "rsa_hostkeys" {                                       
#  for_each  = toset([for x, values in var.instances: values["prefix"]])           
#  algorithm = "RSA"                                                               
#  rsa_bits  = 4096                                                                
#}                                                                                 
                                                                                   
resource "tls_private_key" "ssh" {                                                 
  count     = var.generate_ssh_key ? 1 : 0                                         
  algorithm   = "ECDSA"                                                            
  ecdsa_curve = "P256"                                                             
}                                                                                  
                                                                                   
resource "tls_private_key" "rsa_hostkeys" {                                        
  for_each  = toset([for x, values in var.instances: values["prefix"]])            
  algorithm = "ECDSA"                                                              
  ecdsa_curve = "P256"                                                             
} 

This is related to issue #4.

[victorusu]

Ciao @cmd-ntrf,
I haven't seen your reply before. Thanks for investigating this issue.
I think that you are completely correct. The only workaround that I could find out is to change the type of keys being used.

Since I had seen the #4, I thought about giving it a try, even just in a hacky way.

I think that in order to bypass the issue that you have pointed out, we should work on getting the #4 fixed.
Do you want to have to support both keys, or do you want to replace RSA with ECDSA?

Thanks again.

[cmd-ntrf]

I dropped support for ECDSA because I was hopeful we would be able to Ed25519 (which eventually and in the mean time supporting only RSA was fine.

There is not much we can do about #4. It is up to the Terraform TSL provider to make it work. The issue has been raised a few times: https://github.com/hashicorp/terraform-provider-tls/issues, but nothing looks close to getting merge upstream.

Since this appears to be specific to Fedora 34, I do not see much arguments to integrate ECDSA for now. Unless you think supporting Fedora 34 would strongly benefit Magic Castle users?

[victorusu]

Thanks, @cmd-ntrf for the info. I was not aware of it.
I agree with you. I think that we should wait. There is no real benefit to use Fedora 34 as a support Linux distro. Rocky, CentOS, and RedHat should be the way to go.

Thanks again!