Parse netrc file when downloading test files

[dbaston]

Expected behavior and actual behavior.

I would expect the following function call to pull my EarthData login from my .netrc file, since authentication is required to download the file.

gdal/autotest/gcore/hdf4_read.py

Line 267 in 34c6d2f

if not gdaltest.download_file('https://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.006/2006.06.10/MOD13Q1.A2006161.h34v09.006.2015161173716.hdf', 'MOD13Q1.A2006161.h34v09.006.2015161173716.hdf'):

Python provides a netrc module that will do this, though it's not straightforward to use in this case because the download URL does a 302 redirect to urs.earthdata.nasa.gov (which would have the .netrc entry) and then back to the USGS server. So we can't directly recognize that we have a login available from parsing the base URL.

An alternative might be to add a password manager (as in the NASA instructions) and populate it with the contents of the user's .netrc file.

Another alternative: avoid downloading test files that are behind a login?

Steps to reproduce the problem.

Operating system

Pop OS 20.04 64 bit

GDAL version and provenance

master

[rouault]

avoid downloading test files that are behind a login?

That would good. We have the possibility to store files at https://download.osgeo.org/gdal/data/ , but we need to make sure that the license of the source data is known, and the file size must be not too big as storage capacity on osgeo server is limited.