http: migrate to fsspec

[isidentical]

http is an important remote, and we should migrate it to fsspec. There is already a built-in http filesystem inside fsspec, but it seem to lack a couple of features we depend. So there should be some benchmarkings to determine whether we should create a sync, requests backed httpfs or just use the HTTPFileSystem from fsspec.

[isidentical]

It seems like with some changes (see fsspec/filesystem_spec#731) passes all the functional tests against HTTP. I am now moving to the phase of running benchmarks, but some early microbenchmarks indicate that as is aiohttp is not as performant as requests when it comes to the threaded code. Still need to do more concrete examples (thinking about #6222 using as a benchmark case).

[isidentical]

I might've spoke a bit too soon, seems like it was causing due to a weird race which now is fixed but still it is not outperforming, but just catching up;

http://speedtest-ny.turnkeyinternet.net/1000mb.bin (get-url)

    regular: 4m9.420s
    httpsfs: 4m1.616s

dataset-registry/mnist/images (pull)

    regular: 19m0.577s (1 fail)
    httpfs: 18m54.310s (no fail)

[skshetry]

but still it is not outperforming, but just catching up;

@isidentical, did you check with HTTPStreamFile? We stream download the file using IterStream, whereas default open in fsspec.http uses chunking.

dvc/dvc/utils/stream.py

Line 11 in 13d1276

class IterStream(io.RawIOBase):

---

EDIT: Maybe not, I see that it has fetch_all that gets everything in one go.
Also, the differences are quite small, could be chunking or any other things.

[isidentical]

Also we directly use get_file / put_file so I don't think HTTPStreamFile affects anything in the benchmarks.