Machine readable form of attributes

[lhmarsden]

Question

Is there a machine-readable form of the attributes somewhere? Perhaps in XML or similar.
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.11/cf-conventions.html#attribute-appendix

I see that these are not considered a 'vocabulary' in the same way that the CF standard names etc are.

I ask because I developed with @filchos a spreadsheet template generator with a configuration for CF-NetCDF. It is designed to help people who prefer to work usually with spreadsheets get their data into some more structured format before they start converting them. The user can select column headers from the full list of CF standard names, and there is a separate sheet for global attributes.

I would like to add a separate sheet for variable attributes too at some point. Hence this request.

The template generator is hosted here:
https://www.nordatanet.no/aen/template-generator/config%3DCF-NetCDF

The code is here (anyone reading is welcome to provide feedback using issues or we can work together on developing it further)
https://github.com/SIOS-Svalbard/Nansen_Legacy_template_generator

We published a paper on this over the summer:
https://doi.org/10.5334/dsj-2024-038

It might be that we work on a converter to CF-NetCDF at some point, probably starting with some low hanging fruit e.g. time series or vertical profiles.

[feggleton]

Last year I added the CF Attributes from the appendix into the NVS, they live here https://vocab.nerc.ac.uk/collection/P31/current/

Please be aware that there currently isn't an update process for this vocabulary but I would like there to be. As such it's static from the date of being published and hasn't been updated since then. If you'd like it update and something has changed, I can do that for you.

[lhmarsden]

That's great, @feggleton!

Do you know how I would read an NVS vocabulary into Python without downloading the XML or JSON file first? Ideally I want a URI so that I can run my script again later to pull in any updates that could be made to the vocabulary later.

[lhmarsden]

It is okay, I can use this
https://vocab.nerc.ac.uk/collection/P31/current/?_profile=nvs&_mediatype=application/ld+json

[lhmarsden]

Do you know if it is possible to add whether attributes can be used for variables, global attributes etc? Or would I have to hard code this information

[lhmarsden]

If it isn't possible/appropriate to store this information on NVS, we/I could perhaps write something to read this into a JSON/XML file we could host in GitHub that could include this information. Then we could also populate the table in the conventions document based on that. This was, the NVS vocab would be the source we should edit and changes will propegate through.

I will probably need to write this for myself in any case.

What do you think?

[larsbarring]

Hi Luke @lhmarsden
I do sympathize with your need to programmatically use and process the list of attributes. But this list is part of the CF conventions, and I think it would be cumbersome if we were to (caveat emptor: I have misunderstood what you suggest, in that case my apologies):

1. discuss enhancements and changes in Discussions,
2. dig into more concrete word-smithing include changes to appendix A in the cf-conventions repo,
3. then extract those changes and get them into NVS table,
4. then use that table to populate the appendix A table,
5. then to make sure everything is as we expected.
6. if not, go back to 2. or 3.

Perhaps it is more straightforward to do it the other way around (which essentially is what Fran manually did):
After every new CF conventions release (not the drafts!) programmatically extract the Appendix A table information from the appa.adoc source into a json/xml/yaml/whatever-suitable and ship it over to NVS for updating the table. The the CF community (we) will as usual focus on points 1, 2, 5.

[sethmcg]

Let me propose a different solution:

The CF Conventions are written in asciidoc, and the table in Appendix A has a very regular structure. Rather than going the roundabout route via XML/JSON from the NVS, it shouldn't be very hard to parse the original asciidoc file directly.

I don't know if you can find a library in Python that will do it out of the box -- the existing packages are more focused on providing the full set of machinery to convert asciidoc into HTML & PDF, whereas here all you want is to extract the contents of the table in an array-like form -- but all you need is code that can detect the beginning and end of the table (|=====), beginning of a new cell (|), and get the number of columns from the cols field in the [[table attributes]] block at the beginning.

So if you've got that, you can point the script at the relevant file in the github repo to get the most up-to-date form, and we don't have to add the extra files / steps that Lars is concerned about.

And I think then that the info Luke is looking for about whether attributes can show up as global or on a variable, etc., is already there in the "Use" column of the table.

[cofinoa]

A intermediate way could be use a TSV file, to be include in the ADOC file:

[%header,format=tsv]
|===
include::attributes.tsv[]
|===

See AsciiDoctor doc at:
https://docs.asciidoctor.org/asciidoc/latest/tables/data-format/#delimiter-separated-values

[lhmarsden]

I like the idea of @cofinoa of using a TSV file as the source for this table in the asciidoc. It would make it much easier for people to use than trying to extract these data from the asciidoc. This approach would be more robust.

Whilst I am sure it is not too much work to pull data from the asciidoc file, any code I would write could break if the asciidoc changes.

Shall I work on creating this TSV and updating the asciidoc to read this, and submit a pull request?

[JonathanGregory]

Dear Luke

Since this would be a change to the conventions document, I think you should propose it as an enhancement issue in the conventions repo, and link the pull request to that issue. Thanks for raising it.

Cheers

Jonathan