The progress=true option breaks upload via an Nginx reverse proxy #6402
Comments
|
When IPFS has been stopped for more than 10 minutes, could you run |
|
IPFS is waiting for more data. Some HTTP implementations don't like it when the server responds before it reads the entire body. I know we had to work around this in go in ipfs/go-ipfs-cmds#116. When you say "The nginx proxy passes a chunk of data", is that all the remaining data? Trace: |
|
What is the status of this bug now? |
|
Waiting on more information. From the stack traces, go-ipfs looks like it's operating correctly. If you want to try debugging this, try using wireshark to dump the http request/response between nginx and go-ipfs. |
|
@johanherman did some research about this problem:TLDR:
Note: we have report saying that using
|
|
From my tests with wireshark, nginx stops sending content to ipfs from the very moment it receives the first response chunk (the progress object). When and how that happens is controlled by the proxy buffering options in nginx so it may looks like it works for a bit and then stalls. I don't think we can do much about this. |
|
Hi everyone, I first noticed the issue while using ALB, but then I switched to NLB to simply proxy TCP connection to a target group. I dug deeper since I wanted to enable SSL, so I put Nginx in front of each IPFS node to expose it using HTTPS and got progress option via disabling buffer and enabling HTTP/1.1 for proxying. |
|
Thanks for solving this! I've added it to some documentation in #7469. |
|
I tried the solution suggested by @dsemenovsky and it does not seem to help: I'm running |
|
Hey @jakubgs can you post your nginx config (location section) and tell me what other load balancers you have in front of nginx? |
|
No other lbs, as you can see I'm querying port My location /api/v0/add {
proxy_pass http://localhost:9095;
}The settings you suggested were put under |
|
Try having only one section that proxies all paths to your IPFS, and try putting everything in the location section. That’s what I did, and if it works for you too it would be easier for you to debug once you start moving directives to /api/v0/add location one by one. |
|
It's pointless to try that since the settings are marked as affecting both |
|
Sorry it didn’t. |
|
Yeah, it's tricky. And just to be clear, I tried all 6 combinations of the options, directly in the It did work once with just |
|
By the way, I am using Nginx 1.16 if that helps.
IPFS is v0.5.1 same as yours.
—
Dmitry Semenovsky
+1 (416) 455-0212
… On Jun 15, 2020, at 10:20 AM, Jakub ***@***.***> wrote:
Yeah, it's tricky. And just to be clear, I tried all 6 combinations of the options, directly in the location field.
It did work once with just proxy_buffering, but I couldn't reproduce it, and it does sometimes work with smaller files(~7-8MB) without good reason and randomly.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
@Stebalien instead of streaming progress updates from go-ipfs to the client, would it be possible to compute the progress client-side only by metering how much data has been received ? It would avoid having this sort of bidirectional HTTP request which is at least a gray area if you read the RFCs. |
For just progress, yes. But But yes, clients that can't handle bidirectional streaming can implement client-side progress. |
Stebalien
added
topic/docs-ipfs
|
Why was this closed? Did anyone confirm the fix actually works? |
|
It was closed automatically because I merged a linked issue. However, it's not a bug in go-ipfs itself so it really doesn't belong in the issue tracker (and we can't do anything about it other than try to help you debug yoru setup). If you're looking for more help on this, you should try reaching out on the forums and/or IRC/Matrix. |
|
|
|
Well, you've prompted me to dig into this a bit more. I assumed that @dsemenovsky's fix worked, but I couldn't replicate it. A bit of searching turned up:
Assuming these are up to date, I'm not sure what we can do other than to ask upstream to fix the bug (or someone could submit a patch). We're planning on eventually replacing the API with something better (probably a websocket based CBOR-RPC transport), but that's not a short term solution. Note: this isn't just about
For the moment, I can do is add some documentation explaining why this doesn't work. But it's not going to be a "quick fix" unless someone can convince NGINX to fix the issue on their end. @MichaelMure have you never run into this in production? |
|
@Stebalien we do and are experimenting to find a solution. Notably, AWS ALB (layer 7) seems to be picky about this, we are going to migrate over NLB (layer 4). It will likely not be enough though. My understanding is that (for a good reason or not), a lot of networking code around there assume that a HTTP response can only come after the request has been fully received by the server. It's also the case in |
|
Yeah, we've run into the |
|
@dsemenovsky's fix worked for me with |
|
It's worth noting that @dsemenovsky was on Nginx |
|
Hm. I tried the fix on 1.18 and 1.19 and it failed with a large (10GiB) directory: worker_processes 1;
events {
worker_connections 1024;
}
http {
default_type application/octet-stream;
server {
listen 127.0.0.1:80;
server_name localhost;
location / {
proxy_request_buffering off;
proxy_buffering off;
proxy_http_version 1.1;
client_max_body_size 100G;
proxy_pass http://localhost:5001;
}
}
} |
|
I've also tried inserting socat between nginx, the go-ipfs client, and the go-ipfs daemon. No dice. |
|
Used proposed solution. Got an error After using ipfs.files.add:
Hash still returns but only reachable through same node as uploaded from. IPFS Gateway returns a 502. |
|
I've upgraded my IPFS cluster hosts to Ubuntu > nginx -v
nginx version: nginx/1.18.0 (Ubuntu)
> sudo grep -A 3 'Fix' /etc/nginx/sites-enabled/ipfs.example.org.conf
# Fix uploads of big files
proxy_request_buffering off;
proxy_buffering off;
proxy_http_version 1.1;
> dd if=/dev/urandom of=file.txt count=10 bs=1M
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.114654 s, 91.5 MB/s
> curl -k --max-time 10 -F '[email protected]' -H 'host: ipfs.example.org' 'https://localhost:443/api/v0/add?progress=true'
{"Name":"","Hash":"b","Bytes":262144}
{"Name":"","Hash":"b","Bytes":524288}
{"Name":"","Hash":"b","Bytes":786432}
{"Name":"","Hash":"b","Bytes":1048576}
{"Name":"","Hash":"b","Bytes":1310720}
{"Name":"","Hash":"b","Bytes":1572864}
{"Name":"","Hash":"b","Bytes":1835008}
{"Name":"","Hash":"b","Bytes":2097152}
{"Name":"","Hash":"b","Bytes":2359296}
{"Name":"","Hash":"b","Bytes":2621440}
{"Name":"","Hash":"b","Bytes":2883584}
{"Name":"","Hash":"b","Bytes":3145728}
{"Name":"","Hash":"b","Bytes":3407872}
{"Name":"","Hash":"b","Bytes":3670016}
{"Name":"","Hash":"b","Bytes":3932160}
{"Name":"","Hash":"b","Bytes":4194304}
{"Name":"","Hash":"b","Bytes":4456448}
{"Name":"","Hash":"b","Bytes":4718592}
{"Name":"","Hash":"b","Bytes":4980736}
{"Name":"","Hash":"b","Bytes":5242880}
{"Name":"","Hash":"b","Bytes":5505024}
curl: (28) Operation timed out after 10000 milliseconds with 816 bytes received |
|
@ryanshahine you have a different issue. Please ask on https://discuss.ipfs.io. @jakubgs I can confirm. I can't get this working on any version of nginx. |
|
Managed to get this work by adding
|
I fail to see how this would do anything. This setting is not related. Likely it just works by chance because the request body upload finishes before the response starts being written. Note the error usually appears sporadically. |
Solved my problem, can toggle it on and off by adding or removing that line. |
Version
I'm using a docker container: `ipfs/go-ipfs:v0.4.20``
Description
When using the
progress=trueflag with the/api/v0/addAPI path the upload just stops midway for files larget than a few MB(the larger the more likely to stop).Steps to reproduce
5001:progress=trueenabled:And then it just stops, indefinitely.
If the
progress=trueoption is omitted everything works just fine.Details
Naturally the upload works just fine without the proxy. I've tried watching the traffic with
tcpdumpto figure out who stops responding and it seems to be the IPFS service. The nginx proxy passes a chunk of data, and then IPFS doesn't respond, and everything stops.The text was updated successfully, but these errors were encountered: