Fix DAP4 remotetest server #2558
Conversation
NOTE: This PR should not be included in 4.9.1 since additional
DAP4 related PRs will be forthcoming.
This PR makes major changes to libdap4 and dap4_test driven by changes to TDS.
* Enable DAP4
* Clean up the test input files and the test baseline comparison files. This entails:
* Remove a multitude of unused test input and baseline data files; among them are dap4_test/: daptestfiles, dmrtestfiles, nctestfiles, and misctestfiles.
* Define a canonical set of test input files and record in dap4_test/cdltestfiles.
* Use the cdltestfiles to generate the .nc test inputs. This set of .nc files is then moved to the d4ts (DAP4 test server) war file in the tds repository. This set then becomes the canonical set of DAP4 test sources.
* Scrape d4ts to obtain copies of the raw streams of DAP4 encoded data. The .dmr and .dap streams are then stored in dap4_test/rawtestfiles.
* Disable some remote server tests until those servers are fixed.
* Add an option to ncdump (-XF) that forces the type of the _FillValue attribute; this is primarily to simplify testing of fill mismatch.
* Minor bug fixes to ncgen.
* Changes to libdap4:
* Replace old checksum hack with the dap4.checksum flag.
* Support the dap4.XXX controls.
* Cleanup _FillValue handling, especially var-attribute type mismatches.
* Fix enum handling based on changes to netcdf-java.
* Changes to dap4_test:
* Add getopt support to various test support programs.
* Remove unneeded shell scripts.
* Add new scripts: test_curlopt.sh
Warning: This PR is a follow on to PR Unidata#2555 and should not be merged until that prior PR has been merged. The changeset for this PR is a delta on the PR Unidata#2555. This PR re-enables the use of the server *remotetest.unidata.ucar.edu/d4ts* to test several features: 1. Show that access over the Internet to servers using the DAP4 protocol works. 2. Test that DAP4 support in the [Thredds Data Server](https://github.com/Unidata/tds) is operating correctly. 4. Test that the DAP4 support in the [netcdf-java library](https://github.com/Unidata/netcdf-java) library and the DAP4 support in the netcdf-c library are consistent and are interoperable. The test inputs (primarily *\*.nc* files) provided in the netcdf-c library are also used by the DAP4 Test Server (aka d4ts) to present web access to a collection of data files accessible via the DAP4 protocol and which can be used for testing Internet access to a working server. To be precise, this version of d4ts is currently in unmerged branches of the *netcdf-java* and *tds* Github repositories and so are not actually in the main repositories *yet*. However, the *d4ts.war* file was created from that branch and used to populate the *remotetest.unidata.ucar.edu* server The two other remote servers that were used in the past are *Hyrax* (OPenDAP.org) and *thredds-test*. These will continue to remain disabled until those servers can be fixed. ## Primary Changes * Rebuild the *baselineremote* directory. This directory contains the validation data needed to test the remote servers. * Re-enable using remotetest.unidata.ucar.edu as part of the DAP4 testing process. * Fix the *dap4_test/test_remote.sh* test script to match the current available test data. * Make some changes to libdap4 to improve the ability to catch malformed data streams [affects a lot of files in libdap4]. ## Misc. Unrelated Changes * Remove a raft of warnings, especially in nc_test4/tst_quantize.c. * Add some additional explanatory information to the NCZarr documentation. * Cleanup some Doxygen errors in the docs file and reorder some files.
|
This pull request introduces 2 alerts when merging 591e6b2 into 06df257 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
|
This pull request introduces 2 alerts when merging d06901f into 06df257 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
|
Note on the LGTM suppression. The code tries both HTTP and HTTPS |
|
@DennisHeimbigner thanks for the heads up. Also, I'm bumping the item I have to migrate to Github code scanning, since the Dec. 5 deadline is coming up fast. |
|
This pull request introduces 2 alerts when merging 65eca29 into 0fc2c81 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
|
This pull request introduces 2 alerts when merging 7fd059f into 0fc2c81 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
This change-set modifies PR Unidata#2555 to add the changes listed below. Most of these changes are required by changes to the Java remotetest.unidata.ucar.edu server. ## DAP4 Related Changes * Add tests *dap4_test/test_constraints.sh* and *dap4_test/test_hyrax.sh*. * Provide explicit list of remotetest files to test. * Cleanup local checksum computing and verification. * Define a temporary Hyrax hack flag to deal with the way Hyrax handles checksums and add "#hyrax" fragment flag for it. * Add a hack to get past an LGTM problem with using "http:". * Improve debug support. ## Other Changes * Cleanup the recipe in *docs/nczarr.md* for building *aws-sdk-cpp* library.
|
Added some additional changes in commit d1d2808. |
Warning: This PR is a follow on to PR #2555 and should not be merged until that prior PR has been merged. The changeset for this PR is a delta on the PR #2555.
This PR re-enables the use of the server remotetest.unidata.ucar.edu/d4ts
to test several features:
The test inputs (primarily *.nc files) provided in the netcdf-c library
are also used by the DAP4 Test Server (aka d4ts) to present web access to a
collection of data files accessible via the DAP4 protocol and which can be
used for testing Internet access to a working server.
To be precise, this version of d4ts is currently in unmerged branches
of the netcdf-java and tds Github repositories and so are not actually
in the main repositories yet. However, the d4ts.war file was created
from that branch and used to populate the remotetest.unidata.ucar.edu
server
The two other remote servers that were used in the past are Hyrax (OPenDAP.org)
and thredds-test. These will continue to remain disabled until
those servers can be fixed.
Primary Changes
Misc. Unrelated Changes
Addendum (1/18/2023)
Additional DAP4 fixes
This change-set modifies PR #2555
to add the changes listed below. Most of these changes are required
by changes to the Java remotetest.unidata.ucar.edu server.
DAP4 Related Changes
Other Changes