How to pass ssh key to dagu step?

[alext-extracellular]

Hi there, I am trying to run an rsync step in dagu in order to back some files up.
The rsync needs to connect to a remote host using ssh. I am running dagu with docker compose.
I am mounting my host keys onto the dagu scheduler container ( I also tried the dagu server one earlier) and then inside my DAG, I am referencing the path of the key inside the container. However, the rsync step is failing due to the key not existing in that location.
So I am wondering, how is dagu running the steps? how can I provide the ssh key file to the process?

Thanks for your help

[yohamta]

Hi, thanks for providing the details. I suggest using environment variables to pass the SSH key. Here’s how you could set it up:

env:
  - SSH_KEY=<ssh key>

steps:
  - name: setup ssh-key
    command: sh
    script: | 
      # setup SSH credentials using $SSH_KEY
  - name: rsync
    command: [rsync, <options>]
    depends_on: setup ssh-key

Does this solve your issue? If it doesn't, please feel free to share more details so I can assist you further.

Note: the env key setting for SSH_KEY isn't actually necessary in the DAG if the environment variable is provided to the process.

[alext-extracellular]

Aha, so I can put the ssh public and private keys in environment variable of the container, and then write them to the file in the DAG before the rsync is executed. I will five this a go, thank you

[alext-extracellular]

So it seems the issue was not with how I was passing the SSH keys in, but with how I was running the rsync command. I tried with the environment variables and it still had the same problem. Then I tried with writing the command differently and that worked with no issues.

Here is the step that was not working:

  - name: run rsync
    depends:
      - install rsync
      - install ssh
      - setup ssh
    command:
      - "rsync"
      - "-avP"
      - "-e"
      - '"ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /home/dagu/.ssh/id_ed25519"'
      - "--exclude"
      - ".ssh"
      - "--dry-run"
      - "user@host:/home/user/"
      - "/path/to/data"

This gave the error:

rsync: [Receiver] Failed to exec ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/dagu/.ssh/id_ed25519: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(85) [Receiver=3.3.0]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in IPC code (code 14) at io.c(231) [Receiver=3.3.0]

This made me think the file was not there, but it actually was, just something was wrong with rsync.

Then I changed the format to this:

 name: run rsync
    depends:
      - install rsync
      - install ssh
    command: sh
    script: |
      rsync -avP -e "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /home/dagu/.ssh/id_ed25519" --exclude .ssh --dry-run user@host:/home/user/ /path/to/data

Maybe it is to do with how the command and arguments are passed when it is written as a list?
Either way, thank you for the help.

[alext-extracellular]

For clarity, you can pass the ssh keys either by mounting them onto the dagu server container, or by using environment variables as @yohamta shows in their answer. Either will work, and the keys will be available to the DAG steps.