Copying files over TRAMP

[jumper047]

Dirvish's file copying mechanics sadly don't work over tramp, while dired default copying/moving works fine.

[alexluigit]

I am planning to bring in rsync to deal with copy/paste between remote hosts. But it might takes me some time to implement this, because the dirvish-yank library needs some refurbishing. See #34 (comment) and #26

[alexluigit]

Please checkout the latest commit, I don't have time to test it for now, so any feedback/bug report is appreciated. Cheers.

[noctuid]

I could not get it to work over ssh. For dirvish-yank or dirvish-move with just TRAMP/ssh, nothing happens. If I use both sudo/ssh with TRAMP, I get an error:

Error running timer: 'dirvish-yank--status-update': (remote-file-error "Forbidden reentrant call of Tramp"

Edit: To clarify I only tried moving files on the remote host not from the remote to the local or the other way around. Should both work?

[alexluigit]

I have ssh config looks like this.

TCPKeepAlive yes
AddKeysToAgent yes
AddressFamily inet

Host alex
  HostName myhost.name
  User root
  Port port_number
  IdentityFile ~/.config/ssh/id_rsa

This is how dirvish-yank command work on my machine (I open the connection using find-file).

Peek.2022-05-06.22-11.mp4

remote to local / local to remote works in a similar way (just without overwrite prompt because rsync handles that)

I haven't tested copying files from remote-A to remote-B, not sure if it works.

Does the file preview works over TRAMP? Do you have any special config for TRAMP or ssh?

[alexluigit]

fa5f347 makes the "*Dirvish-yank*" buffer (a hidden buffer, prefix a space to access it) logs the recent 10 yank process. Anything interesting in this buffer? @noctuid

[jumper047]

Just tested with kubernetes pod (via kubel), no luck - copying not working.
Got this error:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  dirvish-yank--status-update()
  apply(dirvish-yank--status-update nil)
  timer-event-handler([t 25206 16955 449155 0.1 dirvish-yank--status-update nil nil 878000 nil])

Also no Dirvish-yank buffer appears

[alexluigit]

Also no Dirvish-yank buffer appears

This buffer won't popup automatically.

(switch-to-buffer dirvish-yank--buffer)

[alexluigit]

I have no idea why dirvish-yank--status-update would raise an error. But since it has nothing to do with the file I/O, for debugging purpose, it's OK to temporarily disable it, just eval this to redefine the dirvish-yank--execute function:

(defun dirvish-yank--execute (cmd)
  "Run yank CMD in subprocesses."
  (let ((process-connection-type nil)
        (procname (format "*Dirvish-yank @ %s*" (current-time-string))))
    (start-process-shell-command procname dirvish-yank--buffer cmd)))

Then try dirvish-yank again and see what's in dirvish-yank--buffer

(switch-to-buffer dirvish-yank--buffer)

[noctuid]

For me, it is some permissions issue. This shows up instead of preview and in the yank buffer logs. Normal dired operations work correctly. Is some extra setup required for dirvish? I do not have ssh-askpass installed on either machine. Should it be necessary?

ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory
<username>@192.168.2.2: Permission denied (publickey,password)

I am connecting to a local computer over ethernet with a password only. I have no ssh or TRAMP configuration.

For whatever reason, the username shown at the end is not the one I used to connect to the remote host with; it is my local username (not sure if that is a problem).

[alexluigit]

This is the excerpt from dirvish.el that in charge of the file preview on remote host.

(dirvish-define-preview remote (file)
  "Preview files with `ls' or `cat' for remote files."
  (when-let ((local (file-remote-p file 'localname)))
    `(shell . ("ssh" ,(file-remote-p file 'host) "test" "-d" ,local
               "&&" "ls" "-Alh" "--group-directories-first" ,local
               "||" "cat" ,local))))

What it does is basically composes and issues a shell command looks like this:

ssh foo test -d /path/to/file && ls -Alh --group-directories-first /path/to/file || cat /path/to/file

The test -d ... ls ... cat ... part in the command is a normal shell command, nothing special.
The ssh foo here makes the command being run remotely instead of locally, where the host name "foo" should be defined in the ~/.ssh/config like

# other configs for performance
# ...
Host foo
  HostName foo.name # in the form of "google.com"
  User root # the login username
  Port port_number # usually 22
  IdentityFile /path/to/identity_file

If I run this command in the terminal, I'll get the text content or directory listing for that file, which is the same content being displayed as preview inside of dirvish.

In other words, if this shell command cannot work properly, the preview cannot work either. The same principle applies to dirvish-yank and friends within the same remote host
(hostA-hostB are handled by rsync, no ssh involved), because they do similar things except using cp or mv and the like to achieve different tasks.

I do not have ssh-askpass installed on either machine. Should it be necessary?

All I installed is the openSSH on Arch, nothing else. Other than that, I've copied the pubkey to the "foo" using ssh-add so that I don't have to enter the password. FWIW I also used ssh-agent to start my X environment, but I'm not sure if that is relevant to this issue.

I did not even tried to use TRAMP for file preview and copy/paste because it seems that there are endless amount of synchronous chitchat on just opening a file. Maybe I'm wrong about that.

Will the ssh config above solves this issue for you? If not, what's the error message in the preview buffer?
I guess this problem can be solved by composing the shell command in a different way or something, but I have no idea on how to do that since I don't have the environment as you described. A workable shell command (without prompting for password) would be very helpful for this issue.

[noctuid]

Will the ssh config above solves this issue for you? If not, what's the error message in the preview buffer? ... A workable shell command (without prompting for password) would be very helpful for this issue.

I will try next time I get the chance. TRAMP does not require any ssh config file though. Shouldn't it be possible to get all this information from Emacs (except maybe the password) and then run ssh -l <user> <host> ...?

If possible, I think it would be better to do things through TRAMP though. It can prompt for the password itself. More importantly, it supports a lot more than just ssh e.g. sudo, docker, and combining different protocols. For example, I used to occasionally pipe ssh to sudo to docker to work inside docker containers on a remote machine (though in this case I was using eshell mostly rather than needing dired). Since dirvish is not using TRAMP, that probably explains why it did not work with a kubernetes pod for @jumper047 (not sure if you can acually ssh into a kubernetes pod or if it works like it does for docker in docker-tramp.el with a separate protocol).

I haven't looked at how TRAMP and dired work together by default, so maybe it is just not easily possible to have async file movement this way. If the current implementation is the only way to do it, the documentation should mention it only supports ssh and requires ssh configuration to avoid asking for a password. I will try to get this working sometime next week. Thanks for the explanation.

[alexluigit]

Shouldn't it be possible to get all this information from Emacs (except maybe the password) and then run ssh -l ...?

This would be a promising solution, but I'm sure if the sub-process have access to the password that is captured by TRAMP, which was the reason I didn't tried this approach. I'll give it a try.

If possible, I think it would be better to do things through TRAMP though.

It would be great if we can do these through TRAMP, that's also the spirit of dirvish. I'll see if I can bypass the synchronous part of TRAMP and come with up a better solution than the current ssh one. I've never been able to read through the source code of TRAMP, because I was always intimidated by the intensive usage of regex there 😕 . But I'll try my best.

If the current implementation is the only way to do it, the documentation should mention it

Definitely. I'll improve the doc on this once I (hopefully) figure out the TRAMP stuff. Thanks for your advice and feedback.

[noctuid]

It would be great if we can do these through TRAMP, that's also the spirit of dirvish. I'll see if I can bypass the synchronous part of TRAMP and come with up a better solution than the current ssh one. I've never been able to read through the source code of TRAMP, because I was always intimidated by the intensive usage of regex there confused . But I'll try my best.

tramp-handle-shell-command supports running async shell commands, though I do not not if that is the correct thing to use for this functionality. I've seen discussions about wanting async file copying with TRAMP without any solutions, so it may be that it is very hard to do with TRAMP.

[alexluigit]

tramp-handle-shell-command supports running async shell commands, though I do not not if that is the correct thing to use for this functionality.

Yes, it probably is the right way to go, I have tried to implement the preview part with this function yesterday and it's working pretty well (although some TRAMP config are required to ensure a usable performance). Since we can use it for preview then I guess we can use it for copy/paste as well. The tricky thing I'm facing now is how to keep as more as features that work locally also works on TRAMP (hopefully we can remove the dirvish-enable-features-on-remote option). Thanks for suggestion anyway, stay tuned!

[alexluigit]

Bad news:

• In order to get asynchronous preview and copy-paste, we have to use the direct-async-process property for the connection, which has a lot of limitation such as no interactive authentication is allowed, see TRAMP-async.
• It is impossible to make async process for multi-pop connection under current implementation of TRAMP, which means we can't do anything for paths like "/ssh:[email protected]|docker:user@container:/path/to/file", it is always handled synchronously.

Good news:

• After proper configuration, all UI niceties such as attributes and mode line segments (with one or two exceptions) that work locally are available over a TRAMP connection, rendered in a non-blocking way.
• The previous ssh hack has been re-implemented with TRAMP, we can dynamically detect whether a connection has the async support or not, the extra functionalities like preview and copy/paste are only enabled when the requirements are met. For synchronous connections, no errors anymore and everything else work like usual.
• It is possible to achieve async preview and copy-paste for docker containers that are deployed locally by the help of docker exec/cp command, I may implement this when I have the need, but until then, PRs are welcomed.

[noctuid]

Thanks for the update.

In order to get asynchronous preview and copy-paste, we have to use the direct-async-process property for the connection

Is it actually impossible, or is the performance just worse? That heading is titled "Improving performance of asynchronous remote processes." Can't you still use asynchronous processes without direct-async-process, or is the performance too bad?

After proper configuration, all UI niceties such as attributes and mode line segments (with one or two exceptions) that work locally are available over a TRAMP connection, rendered in a non-blocking way.

What about over a multi-hop connection? Will they block or just not work?

It is possible to achieve async preview and copy-paste for docker containers that are deployed locally

I don't personally need this.

the rsync command is always run locally as it requires working SSH authentication which bypasses the password entering to work

So file copy/move is still using rsync directly. My main use cases for dired with TRAMP are:

• local sudo
• multi-hop /ssh: to /sudo:

Can rsync work in the second case given that root login is disabled on the remote host? And what will happen with dirvish in the first case?

I don't know if it's possible, but it would be nice if dirvish could fall back to using TRAMP with synchronous copy/paste in cases that rsync can't handle or if things aren't configured correctly to allow rsync with ssh. I generally only transfer small files (code) to remote hosts, so synchronous vs. asynchronous is not a huge deal for me. It looks like there's no good way to make everything async, but I'd still like to be able to consistently use the same keybindings everywhere for multi-stage copy/paste if possible.

A typical way to bypass the ssh authentication would be using ssh-agent, which see.

It looks like this sentence got cut off in the dirvish-yank docstring.

[alexluigit]

Is it actually impossible, or is the performance just worse?

Sorry for the confusion. I'm not a TRAMP expert so my understanding can be wrong, but here's what I found:
tramp-handle-shell-command can handle a shell command that includes "&" no matter the direct-async-process property is present or not within the connection. But before handling every shell command, we have to go through a synchronous connection phase (the "overhead" described in that page, which takes very long time for a remote connection) unless the connection has the direct-async-process property set to t. In other words, it doesn't matter the command itself is handled synchronously or asynchronously because that is not the bottleneck. The synchronous nature of this connection initialization makes it "impossible" for us to implement an usable file preview and etc.

Can't you still use asynchronous processes without direct-async-process, or is the performance too bad?

As I described above, if the connection does not come with the direct-async-process property, every command takes ~ 1-2s (blocking) to establish the connection, then get spawned. Basically unusable. I think the title of that info node is a little bit vague in the sense that it both introduces the direct-async-process and gives some tips to speed it up (like set tramp-verbose)

What about over a multi-hop connection? Will they block or just not work?

Using direct asynchronous processes in TRAMP is not possible, 
if the remote host is connected via multiple hops. 
In this case, TRAMP falls back to its classical implementation.

If I understood this correctly, I think it means "multi-hop connection = always synchronous"
So I disabled these features on synchronous connections (including mulit-hop ones), it won't block. You'll see "File preview is not supported in current TRAMP connection" in the preview window on this kind of connection.

So file copy/move is still using rsync directly.

Yes, but it's only used when source files and target are located in different hosts (local-to-or-from-remote, remoteA-remoteB).

local sudo

In theory, everything works locally should work for in a local sudo connection, I'll make them available on next update if everything goes fine.

Can rsync work in the second case given that root login is disabled on the remote host? ... multi-hop /ssh: to /sudo:

I'm not sure but I guess it would work because I found this https://github.com/stsquad/dired-rsync/blob/master/dired-rsync-ert.el#L58, I'll have a look on this.

I don't know if it's possible, but it would be nice if dirvish could fall back to using TRAMP with synchronous copy/paste in cases that rsync can't handle or if things aren't configured correctly to allow rsync with ssh.

Good idea. I believe this can be done by the help of tramp and dired-aux. Thanks for this.

[noctuid]

As I described above, if the connection does not come with the direct-async-process property, every command takes ~ 1-2s (blocking) to establish the connection, then get spawned. Basically unusable

So I disabled these features on synchronous connections (including mulit-hop ones), it won't block. You'll see "File preview is not supported in current TRAMP connection" in the preview window on this kind of connection.

I assume this was how long it took on a remote connection? Did you test with a completely local multi-hop connection? I bring this up because my usage of TRAMP has been mostly connecting to a local mini pc directly over an ethernet cable. In this case, the connection might be much faster. I don't know if it would be fast enough to allow preview to be usable (even a small synchronous delay would be annoying), so the current default behavior makes sense, but if it is a lot faster, it might make sense to let the user opt-in to some features for certain synchronous connections (e.g. local multihop sudo|docker or ssh|sudo for a local pc). Missing preview is not a huge deal in these cases though. I care more about copy/paste.

[alexluigit]

I assume this was how long it took on a remote connection?

Yes.

Did you test with a completely local multi-hop connection?

Nope, because I don't have an available local computer, please help me to test with the case C below. If everything goes fine, the expect behavior would be:
A. connections w/o proper ssh config: defaults to dired-copy/rename-file (namely synchronous)
B. connections w/ direct-async-process property: non-blocking
C. local connection (local multi-hop connections not tested): non-blocking

How does the preview and copy/paste work on paths like "/ssh:HOST_ON_ETHERNET|sudo:root@servername:/path/"? Not working at all? Performs bad? If it performs badly, will adding HOST_ON_ETHERNET to dirvish-whitelist-host-regex bring in any improvement?

[alexluigit]

When I open Dirvish over a remote connection for which both users have rsync installed and have "direct-async-process" set to t, Emacs hangs indefinitely on the Dirvish buffer.

@tdstoff , by 'hangs indefinitely' did you mean it hangs after invoking dirvish-yank? It would be nice if you can pin down the cause, maybe by play around with dirvish-yank--l2fr-handler, l2fr means local to/from remote. Thank you.

[tdstoff]

@tdstoff , by 'hangs indefinitely' did you mean it hangs after invoking dirvish-yank?

No, actually. I set the following:

  (add-to-list 'tramp-connection-properties
               (list (regexp-quote "/ssh:myuser@host:")
                     "direct-async-process" t))

and notice that file sizes are then shown like in this screenshot:

But within ~10 seconds or so of using the buffer, I'm stuck in a hang that incessant C-g's can't remedy. No use of dirvish-yank required to get this behavior.

Edit: I reproduced this three times in the same remote directory, then tried accessing the remote host at a different location and it seems to be working at the moment, even dirvish-yank. I can keep fuddling with it and see if there's some consistent cause. If you have any suggestions off the top of your head, they're appreciated.

[alexluigit]

See: #124

[tdstoff]

@tdstoff , by 'hangs indefinitely' did you mean it hangs after invoking dirvish-yank?

No, actually. I set the following:

  (add-to-list 'tramp-connection-properties
               (list (regexp-quote "/ssh:myuser@host:")
                     "direct-async-process" t))

and notice that file sizes are then shown like in this screenshot:

But within ~10 seconds or so of using the buffer, I'm stuck in a hang that incessant C-g's can't remedy. No use of dirvish-yank required to get this behavior.

Edit: I reproduced this three times in the same remote directory, then tried accessing the remote host at a different location and it seems to be working at the moment, even dirvish-yank. I can keep fuddling with it and see if there's some consistent cause. If you have any suggestions off the top of your head, they're appreciated.

Does the above setting "direct-async-process" t for a remote connection break shell/terminal buffers in Emacs for anyone else? After I do this, I get the message stty: standard input: Inappropriate ioctl for device in my shell buffer, and vterm processes immediately die upon spawning.

Is this setting inherently not compatible with shells? I asked over at the vterm repo here but they seem quite busy.