--download reports an "incomplete download" with Content-Encoding: gzip and content-length.

[twopir]

When using HTTPie to download a file which is both gzip'd and has a content-length header sent, the final output report says that the download is incomplete. This is apparently because HTTPie tracks the number of bytes it wrote, not the number of bytes it received.

Here is an example, which reproduces in HTTPie head as of about 10 minutes ago.

$ http --debug --download GET http://torcache.net/torrent/640FE84C613C17F663551D218689A64E8AEBEABE.torrent 
HTTPie 1.0.0-dev
HTTPie data: /Users/pi/.httpie
Requests 2.6.0
Pygments 2.0.2
Python 2.7.10 (default, Oct 23 2015, 18:05:06) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] darwin

>>> requests.request(**{'allow_redirects': True,
 'auth': None,
 'cert': None,
 'data': OrderedDict(),
 'files': DataDict(),
 'headers': {'Accept-Encoding': None, 'User-Agent': 'HTTPie/1.0.0-dev'},
 'method': u'get',
 'params': ParamsDict(),
 'proxies': {},
 'stream': True,
 'timeout': 30,
 'url': u'http://torcache.net/torrent/640FE84C613C17F663551D218689A64E8AEBEABE.torrent',
 'verify': True})

HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: close
Content-Encoding: gzip
Content-Length: 40082
Content-Type: application/x-bittorrent
Date: Fri, 25 Dec 2015 04:39:03 GMT
ETag: "567c9773-9c92"
Last-Modified: Fri, 25 Dec 2015 01:10:11 GMT
Server: nginx

Downloading 39.14 kB to "640FE84C613C17F663551D218689A64E8AEBEABE.torrent"
 -   0.00 %     0.00 B     0.00 B/s  -:--:-- ETA
http: error: Incomplete download: size=40082; downloaded=40260
Done. 39.32 kB in 0.22557s (174.30 kB/s)

Note that size is the value of Content-length, but downloaded is greater than size because of the gzip encoding.

[fbidu]

Yeah, I could validate it here... I'll try to fix it

[twopir]

I think that fix is still vulnerable to a problem where the gzip content is incompletely downloaded, but still expands to greater than the content-length. I don't have a demonstration handy, though.

[fbidu]

hmm yeah, it could still be trouble... I'll try a better fix

[jkbrzt]

https://github.com/jkbrzt/httpie/blob/2b51cb6687df037b94c901d5a3651296fd3cf07c/httpie/downloads.py#L181-L182

[giuscri]

To me, it seems like Response.raw should be used; otherwise HTTPResponse.iter_body will return gzip -d'd chunks.

[hholst80]

Greetings from 2023.
This is still an issue.

[I] root@xps /tmp
# http -d 'https://XXX.blob.core.windows.net/CONTAINER/path/to/processed.json?sp=r&st=2023-10-18T11:45:05Z&se=2023-10-18T19:45:05Z&spr=https&sv=2022-11-02&sr=b&sig=PKZTjGYM0wfeU%2BpF%2FMZ93Y2RxDyVHs1xbOeg4of2nuA%3D'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Length: 2087
Content-MD5: Q6dkiGs7Jcp6PimVCRH09A==
Content-Type: application/json
Date: Wed, 18 Oct 2023 11:45:21 GMT
ETag: "0x8DB4C84489078F7"
Last-Modified: Thu, 04 May 2023 09:45:23 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-blob-type: BlockBlob
x-ms-creation-time: Thu, 04 May 2023 09:45:23 GMT
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-request-id: ce24f842-101e-006a-72b8-01eea9000000
x-ms-server-encrypted: true
x-ms-version: 2022-11-02

Downloading to processed.json
Done. 4.3 kB in 00:0.10676 (39.9 kB/s)

http: LogLevel.ERROR: Incomplete download: size=2087; downloaded=4264


[I] root@xps /tmp [1]
# 

[hholst80]

To me, it seems like Response.raw should be used; otherwise HTTPResponse.iter_body will return gzip -d'd chunks.

I think the output should be the decompressed output as it is now. It is just the warning that should go away. If the raw download matches what the http header said, all is good.

Make raw an opt-in for those cases when you really do not want the raw content encoding to be downloaded as-is. I think this issue should focus on getting rid of the error message by measuring the right thing.