[5.x]: Problem opening NC4/HDF5 file due to data object header magic #823
Labels
Comments
|
Hmmm...this is indeed an interesting case. The root issue is netCDF-Java is struggling with the |
|
I saw the same thing as you - a zero as a reference in a DIMENSION_LIST
object. I dont remember this specific thing, or a workaround.
It would take me a while to spin up more on it, which I wont be able to do
right now. If nccopy repairs it, its certainly an indication of a bug in
the original hdf5 library, now fixed. Or a lucky coincidence.
…On Tue, Sep 14, 2021 at 9:55 AM Sean Arms ***@***.***> wrote:
Hmmm...this is indeed an interesting case. The root issue is netCDF-Java
is struggling with the DIMENSION_LIST attribute of the ricfer_channel_name
dataset (full path /Annotation data/Instrumental
information/ricfer_channel_name). The DIMENSION_LIST holds a reference,
and we are reading the heap address of the reference as 0, which isn't
correct. Interestingly enough, if I copy the dataset using nccopy (nccopy
-k nc4
FLX_GPP__BRK_L1A____20190914T103613_20190914T103615_20210910T112811__18260.HRE1.hdf
rewrite.hdf), things work. It makes me wonder if there was a bug in
netCDF-C or HDF5 (netcdflibversion=4.6.0|hdf5libversion=1.10.0) that was
addressed by rewriting the file (netcdf=4.7.3,hdf5=1.10.4), and perhaps the
netCDF-C library has a workaround in place that allows things to work?
@JohnLCaron <https://github.com/JohnLCaron> - have you seen behavior like
this before?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#823 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAEVZQHE6VGT4BFGDPZT4NDUB5WAFANCNFSM5D7K64HQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
You could check with hdf5 support..
…On Wed, Sep 15, 2021 at 12:21 PM John Caron ***@***.***> wrote:
I saw the same thing as you - a zero as a reference in a DIMENSION_LIST
object. I dont remember this specific thing, or a workaround.
It would take me a while to spin up more on it, which I wont be able to do
right now. If nccopy repairs it, its certainly an indication of a bug in
the original hdf5 library, now fixed. Or a lucky coincidence.
On Tue, Sep 14, 2021 at 9:55 AM Sean Arms ***@***.***>
wrote:
> Hmmm...this is indeed an interesting case. The root issue is netCDF-Java
> is struggling with the DIMENSION_LIST attribute of the
> ricfer_channel_name dataset (full path /Annotation data/Instrumental
> information/ricfer_channel_name). The DIMENSION_LIST holds a reference,
> and we are reading the heap address of the reference as 0, which isn't
> correct. Interestingly enough, if I copy the dataset using nccopy (nccopy
> -k nc4
> FLX_GPP__BRK_L1A____20190914T103613_20190914T103615_20210910T112811__18260.HRE1.hdf
> rewrite.hdf), things work. It makes me wonder if there was a bug in
> netCDF-C or HDF5 (netcdflibversion=4.6.0|hdf5libversion=1.10.0) that was
> addressed by rewriting the file (netcdf=4.7.3,hdf5=1.10.4), and perhaps the
> netCDF-C library has a workaround in place that allows things to work?
> @JohnLCaron <https://github.com/JohnLCaron> - have you seen behavior
> like this before?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#823 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAEVZQHE6VGT4BFGDPZT4NDUB5WAFANCNFSM5D7K64HQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Versions impacted by the bug
v5.x, v6.x, v7.x
What went wrong?
A user developing a data product for ESA was unable to open an NC4/HDF5 data in Panoply (currently based on netcdf-Java 5.4.2) due to an "DataObject doesnt start with OHDR" exception thrown by the
H5objectsclass. The same problem occurs using toolsUI 5.4.2 and the most recent IDV download. (FWIW, the file does open in HDFview, but meh.)I added some diagnostics to
H5objects, and found that the object header that it doesn't like is not "OHDR" but instead the 4-char string "ëHDF". A search of the file using a hex editor indicates that the only place that the latter string occurs is at the very start of the file, and is of course the HDF magic string. So it seems that somehow when NJ tries to read the data object, an offset of 0 is being applied?So although I was originally prone to suggest, "Your file is messed up", I am instead asking if instead the file is merely quirky and it is somehow triggering a bug in netCDF-Java. Unfortunately, this is deep enough voodoo that I am posting to the Github repo for help rather than spend a couple weeks trying to parse the file vis a vis HDF format.
A copy of the problematic file is temporarily here on Dropbox. The dataset variable apparently involved with the exception is
ricfer_channel_name�-- omission of that variable from the dataset results in apps being able to open it.Thx.
Relevant stack trace
No response
Relevant log messages
No response
If you have an example file that you can share, please attach it to this issue.
If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.
Yes
Code of Conduct
The text was updated successfully, but these errors were encountered: