Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why not pack data into unsigned int types? #202

Closed
gfireman opened this issue Jan 18, 2023 · 6 comments
Closed

Why not pack data into unsigned int types? #202

gfireman opened this issue Jan 18, 2023 · 6 comments
Labels
question Further information is requested or discussion invited

Comments

@gfireman
Copy link

gfireman commented Jan 18, 2023

Conventions say:

...the variable containing the packed data must be of type byte , short or int .

Why not ubyte, ushort, or uint?

We typically pack data into unsigned int types.
To be CF compliant we'd have to change types - and then adjust scale_factor and add_offset to recover the range lost to a sign bit.

@gfireman gfireman added the question Further information is requested or discussion invited label Jan 18, 2023
@cofinoa
Copy link

cofinoa commented Jan 18, 2023

When the CF convention was written, the netCDF format only supported signed types. This restriction on data model support, it's named netCDF classic model in the current netCDF data model:
https://docs.unidata.ucar.edu/netcdf-c/current/netcdf_data_model.html#classic_model

Probably, support for new types could be proposed as an enhancement.

P.S.: This out of scope for the CF Convention, but if you are using writing packed data to encode new data, you can apply netCDF4 chunking feature, apply shaving to non-significant bits on the float number, and add the deflate filter, honouring classic data model and CF, without the use of CF packed data feature and using float/double original type.

@gfireman
Copy link
Author

gfireman commented Jan 18, 2023

The CF convention does allow for unsigned integer types, just not for packing float/double into.
Interesting suggestion on using compression instead of scaling. We'll run some tests and see how that works for us.
Thanks!

@cofinoa
Copy link

cofinoa commented Jan 18, 2023

yes, you are right!! The restriction on classic data model was removed on CF-1.9, allowing, among others, some data types from the enhanced data model. The Packed Data sections wasn't updated following that.

@larsbarring
Copy link

larsbarring commented Sep 18, 2023

Can this issue now be closed as answered following the completion of #374 via PR#456?

@JonathanGregory
Copy link
Contributor

Yes, I think this question is now answered.

@JonathanGregory
Copy link
Contributor

Dear Gwyn @gfireman

I'm closing this issue now, because it's been addressed by a recent update to the working version of the convention (that will be released in a month or so). Please see the updated section 8.1. Thanks for asking the question. If it's not completely answered, please feel free to reopen this issue.

Cheers

Jonathan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested or discussion invited
Projects
None yet
Development

No branches or pull requests

4 participants