fastGet/fastPut problems with node v14.2.0

[theophilusx]

There appears to be an issue with fastPut/fastGet with latest node version 14.2.0. Problem does not appear to occur with 14.1.0. Suspect issue is related to recent changes in stream API.

Basic problem is fastGet/fastPut seem to hang indefinitely. I have attached a basic script which works fine under earlier versions of node. Other methods, like put() and get() do not seem to be affected. To make matters worse, the issue appears to be intermittent. I can run the attached script and it will work fine 3 times and then fail on the 4th time. I've tried aainst a remote server as well as a local server running on vbox image with same results, so don't think it is local network issue (plus it works fine with earlier versions of node).

lowlevel-fastput.js.gz

[theophilusx]

Just adding this still seems to be an issue with v14.4.0

[mscdex]

fastGet() and fastPut() don't use streams.

[theophilusx]

OK - the reference to streams was only a guess based on warnings in v14 release notes. Up to and including v14.0 fastget/fastput work reliably. From 14.1 up, they fail most times (need to test multiple times before failure in some cases, but usually fails first go).

[Ekliptor]

I am having the same issue.
FastPut starts to upload a file and then stops after sending a few chunks. My code worked with node v12 before.
My environment:

macOS Catalina 10.15.5
node -v v14.4.0

For me the issue is the concurrency parameter. It seems it gets hung up while transferring if concurrency > 1.
My workaround:

sftp.fastPut(localPath, remotePath, {concurrency: 1, chunkSize: 128*1024, step: function(transferred, chunk, total) {
        // uploaded 1 chunk
        // with concurrency = 32 on a 20MB file this gets called about 6 times and then the process just hangs forever
    }}, function(err){
        // upload done. works only with concurrency 1 on node v14
});

[theophilusx]

Setting concurrency to 1 was going to be my next test as well. I will now do that and report back. Your observations are in-line with mine. Whatever is causing the issue seems to have been introduced between 14.0.0 and 14.1.0.

[theophilusx]

I did some more testing and can confirm that fastput/fastget do seem to work reliably if you set concurrency to 1. Values larger than 1 led to unreliable results where the transfer would hang. This issue was still intermittent. You have to test with different destination file names and multiple times (at least 5 would usually result in at least one failure). I found that if a transfer worked, transferring the same file with the same destination name would work reliably multiple times. Changing the destination filename for each test was more reliable in reproducing the failure. I was unable to reproduce the failure at all using a concurrency setting of 1.

[theophilusx]

Appears the original belief this only affected fastget/fastput was incorrect. Seems the issue also exists with using a sftp.writeStream to upload a file. The issue is intermittent - sometimes it works and sometimes it doesn't. Turning on debug seems to indicate that no status is being returned after a write. Have attached the script I've been using. When the script 'hangs', the following debug output is printed

╭─tim@tim-desktop ~/git/ssh2-sftp-client/tools  ‹master*› 
╰─➤  node ./lowlevel-buffer-put-test.js 200 20 /home/tcross/testServer
DEBUG: DEBUG[SFTP]: Outgoing: Writing OPEN
Uploading buffer of 200k to /home/tcross/testServer/test-200k.dat
DEBUG: DEBUG[SFTP]: Parser: Response: HANDLE
DEBUG: DEBUG[SFTP]: Outgoing: Writing FSETSTAT
DEBUG: DEBUG[SFTP]: Parser: Response: STATUS
DEBUG: DEBUG[SFTP]: Outgoing: Writing WRITE

There is no parser: response STATUS liine, which is what you get when the script works.

The attached script works reliably every time with node version 14.0 or earlier. Failures (hanging) is seen with all versions from 14.1 to 14.7. The issue is intermittent and for some reason, if it works, it will continue to work until you delete all the uploaded files. When it fails, running immediately after killing the scirpt will sometimes work and it might be necessary to run it 4 or 5 times before it fails.

The attached script takes 3 command line argument - initial buffer size, number of runs and remote path to directory where the files are to be uploaded to. The script will upload buffers for number of run times, increasing by 10k each run. Starting with an initial buffer size of 200 was fairly reliable in reproducing the issue. Starting with a size of 100 was reliable and worked every time. e.g.

lowlevel-buffer-put-test.js 100 20 /path/to/remote/dir -> works reliably for me
lowlevel-buffer-put-test.js 200 20 /path/to/remote/dir -> hangs nearly every time

lowlevel-buffer-put-test.js.txt

[p-j]

Might be related to mscdex/ssh2#908 ?

[theophilusx]

Yes, I suspect this is related to that issue. My testing shows the problem only occurs when data being sent is sufficiently large enough and the debug seems to show that no status response is being returned after a 'chunk' has been processed, so no further chunks are sent and everything just hangs. I guess we will have to wait until either the change is reverted or some advice on how to fix it in a backwards compatible manner is provided to enable ssh2/ssh2-streams to be updated. BTW the issue is no longer just related to fastput/fastget, put() is also impacted (not tested get() yet).

[mannharleen]

Any update on fixing this please?

[mscdex]

@mannharleen At this point, ssh2-streams isn't going to be maintained going forward. If you would like to instead test with the ssh2 rewrite (currently just the master branch there) to see if the issue still exists there, that would be great.

[theophilusx]

@mscdex I am the maintainer for ssh2-sftp-client, a module which uses ssh2 and ssh2-streams. Thanks for the update. Is the re-write mature enough for me to start working on updating ssh2-sftp-client? Are there any specific issues/changes your aware of with the re-write which may cause issues or is it mainly just a move of ssh2-streams functionality into ssh2? thanks. mscdex <[email protected]> writes:

…

@mannharleen At this point, `ssh2-streams` isn't going to be maintained going forward. If you would like to instead test with the `ssh2` rewrite (currently just the master branch there) to see if the issue still exists there, that would be great.

-- Tim Cross

[mscdex]

@theophilusx there is more information available here

[theophilusx]

@mscdex Thanks. I'll start looking at moving across soon. Timing is quite good as there are some design decisions I made with ssh2-sftp-client which I wasn't happy with and this will give me an opportunity to fix things. The ssh2-sftp-client module has a fairly extensive test suite, so hopefully I can also provide you with some feedback that can help with your porting as well.

[mannharleen]

Ah! Looks like I asked the question at the right time then.

@theophilusx looking forward to the upgrades. Happy to contribute as well

[theophilusx]

@mannharleen I have crated a new branch in the ssh2-sftp-client repo called version-6, which is where I plan to implement the new version based ont the ssh2 re-write. All I have done so far is remove a lot of the extra error checking I previously added. I wasn't happy with how this was working - while it certainly did help end users because it provided much more detailed error feedback, it had a significant performance hit and caused problems on some sftp servers which were very locked down. It also made the code significantly more complex.

At this point, the branch is just a stripped down version of the 5.x version. I haven't yet made changes to the dependencies to use the git version of the ssh2 re-write. Will probably do that in the next few days.

More than happy to receive PR, suggestions, or anything else.

[mannharleen]

@theophilusx will take a look. Glad we are moving forward quickly!

[obrhaberer]

Are there any news on the testing-side?

[theophilusx]

If you are speaking about ssh2-sftp-client, please see the ssh2-sftp-client repository at https://github.com/theophilusx/ssh2-sftp-client . There has been some progress, but slow (I have been busy with other commitments). If you are talking about the ssh2 re-write, I would check the ssh2 repository.

…

-- Tim Cross