Proposal to use NVS P07 as the canonical source of CF standard names

[japamment]

Topic for discussion

Currently the "official" source of the CF standard names is the XML formatted file, e.g., https://cfconventions.org/Data/cf-standard-names/current/src/cf-standard-name-table.xml. A new version of the XML file is produced every time the standard name table is updated. At the same time as publishing the XML file, identical content is submitted to the NERC Vocabulary Server (NVS) where standard names are collection P07: http://vocab.nerc.ac.uk/collection/P07/current/. The proposal is to move to recognising P07 as the canonical source of standard names.

The points below were raised in a hackathon at the CF 2024 workshop.

Advantages of adopting P07 as canonical source of standard names

• Unique identifiers
• Access to FAIR semantics and mappings
• Machine actionability
• Better functionality can be built on the semantics of the vocabulary (SPARQL endpoint
• NVS provides more formats for each vocab
• Potential for streamlining publication process for new vocabulary versions
• Reduces/removes need to store all the standard name table versions in the github repos and on the CF website (helps with space considerations)

Possible consequences of moving to P07

• Reference to Standard names xml in the conventions document (appendix B?) will need to change to p07 NVS
• We will need to ensure that access to the HTML view of standard names is retained on the CF website
• Need to consider the KWIK index and how this will be generated in the workflow
• We are currently reliant on UDUNITS strings to recongnize legal units - these are not currently implemented in NVS

Practical steps we will need to take

• Make a proposal to modify the conventions document to recognise P07, modifications to appendix B?
• We need to consult the community on consequences of removing the XML and HTML versions of the standard name table - how many people's workflow depends on them, how often do people access previous versions of the standard name table
• Tools, python modules and workflows which rely on these files (cfchecker, cfpython?, cfplot?, cdo?, iris?) (mitigation - Xml could be created automatically from NVS)
• Users might want to propose mappings to other vocabularies - how can they do this?
• Viewing previous versions of the collection on NVS is currently not working, have had it working previously so will need to bring this back
• Investigate UDUNIT-QUDT functionality (https://qudt.org/schema/qudt/udunitsCode) as a mean to linking P06 to UDUNIT CF requirement

[ChrisBarker-NOAA]

Investigate UDUNIT-QUDT functionality (https://qudt.org/schema/qudt/udunitsCode) as a mean to linking P06 to UDUNIT CF requirement

I am particularly interested in the units question -- CF has referred to UDUNITS for units since the beginning (yes?) -- but over that time, UDINITS saw a major upgrade that re-worked the unit database, and, at least as far as I could find there is no nice human readable form, nor a clear definition of what variations are acceptable (e.g. both "second" and "seconds" are accepted). So the Conical source could be the XML in the UDUNITS source -- but we really should have a way to extractg from that something human-readable.

I'm willing to take a look at that, unless someone points us to something I missed :-).

[taylor13]

I get a 404 error when I try to reach the xml file: https://cfconventions.org/Data/cf-standard-names/current/build/cf-standard-name-table.xml

Where does it show the names of aliases on the P07 file ( http://vocab.nerc.ac.uk/collection/P07/current/ )?

Why is the JSON-LD version of the file so complicated. Couldn't we simply host the standard name information in a simpler structure with the keys being the standard names and for each we'd have a few "attributes": canonical units, description, aliases, and a flag indicating whether or not the name has been deprecated?

I'm sure I'm hopelessly naive, but it seems like we should host the standard name information in a form that other tools could build on without learning how to interact with the NERC vocabulary server. Is it wise to tie CF to the support of a single institution (which has been a critically important and reliable partner for so long now), which might at some time be unable to continue with CF. In that event, how hard would it be for someone to take over responsibility for the NERC-hosted vocabulary?

[DocOtak]

I've updated the link to the xml document in the original discussion post.

Aliases are the "deprecated" names and what they map to is in the "Same As" relationship.

The JSON-LD file is so complicated I think for two main reasons, one is that it is doing things that the CF XML file does not do, like map to other locations for the same concepts, map to narrower and broader terms (ontology like). The second is that the file is using JSON-LD which has a bunch of RDF stuff going on (that I'll admit I don't fully grok). I think the extra verbosity of the JSON-LD file is similar to CF metadata in netCDF files, all that extra stuff is what makes the file more useful for generic tools.

Re NVS and stability, they are critical infrastructure for many things like argo and SeaDataNet. And I think our entire (?) vocab team is in their employ, so if NVS is going away I don't think the "where will we host" is going to be top of our problems. That said, nothing they are doing is secret or really proprietary, it's all standard link data RDF stuff so we would just need to host those things someplace else. I think NVS is as stable as any other options we might come up with, including continuing to host everything we do on GitHub,

[taylor13]

Thanks, that's quite helpful. I wasn't aware that the standard name database would allow you to "map to other locations for the same concepts, map to narrower and broader terms". Who is responsible for doing the mapping and are there existing tools that make use of the information?
You're quite right that NVS stability isn't at the top of our worries, although I still think it useful to think about doing things in a some other way that isn't quite so complex. Again, I guess depending on how important we think the stuff in paragraph 1 above is, complexity might be warranted.

[JonathanGregory]

Dear standard names hackathon group

While I recognise the advantages of the NERC vocabulary server, I think we should change things more gradually than implied by this summary. When we discussed this possibility in the CF committee, my understanding was that the proposal was initially only to "designate" the NERC vocab server as the primary repository. The primary one is the one you'd trust in the case of an inconsistency, but mostly there would be no inconsistency. Designating NVS as primary would not change our actual practice significantly, I believe, since the standard names team prepare the updates in the CF editor and publish them to the NERC vocab server and the CF website at the same time.

In the earlier discussions, I believe we understood that we would keep the HTML and XML versions on the CF website, just as they are now. I don't see why we shouldn't do that, even if we don't regard them as primary. If we change the process of preparing updates to use NVS facilities instead, could we not continue to publish to our XML and HTML as well? There isn't a need to change Appendix B if we keep our XML files. Our HTML file is very useful for searching, especially since Abel @bzah made large and widely applauded improvements in its capabilities within the last year. In #296, Andrew @DocOtak and Antonio @cofinoa have between them devised ways in which we could keep all versions of HTML and XML accessible online by generating them on the fly from the GitHub repo. That would solve the the problem with space on GitHub Pages.

The first advantage listed is the unique permanent identifier for each standard name. I agree that is valuable. When Alison and I discussed this with Gwen, some months ago, we agreed that CF would like unique URIs containing the standard name. My memory is that this had formerly been offered, although it isn't now. For instance, as an alternative to https://vocab.nerc.ac.uk/collection/P07/current/CFSN0023, we should be able to use https://vocab.nerc.ac.uk/collection/P07/current/air_temperature. For some purposes that would be more convenient, and it's definitely more CF-like.

If the standard names are stored in the NVS as well as our existing files, we can explore the advantages of NVS without losing any of our current functionality. I'm sure there are advantages, such as you outline. In time we may find that NVS can take things over from us in a satisfactory way; then we can phase out our own. Despite my reservations, I appreciate your exploration of this subject! Thanks.

Best wishes

Jonathan

[DocOtak]

Jonathan, I think you were at the BCP14 Hackathon, it was unfortunate that these conflicted.

I don't think any timelines were discussed and the hackathon only tried to identify advantages/disadvantages/issues and the only decisions were to explore solutions or mitigations to the concerns you have raised (they all came up at the hackathon IIRC). I volunteered to explore creating the CF standard name XML using only the information available in the NVS.

Re the alternative to https://vocab.nerc.ac.uk/collection/P07/current/CFSN0023 it is this: https://vocab.nerc.ac.uk/standard_name/air_temperature/ but the preferred labels aren't actual standard names. One of the things we also tried to identify at the hackathon were areas for improvement in the NVS itself if it didn't meet our needs, and this falls under that I think.

[JonathanGregory]

Thanks, Andrew @DocOtak, and for your reply to Karl @taylor13. That's very helpful explanation. Also, I'm glad that NVS does have unique URIs already for individual standard names. We could mention that fact on the CF Vocabularies page.

[gwemon]

@DocOtak @JonathanGregory I think that a point worth bearing in mind is that for optimum reliability and functionality it would be best to recommend the use of standard names of the form http://vocab.nerc.ac.uk/collection/P07/current/CFSN0023 which has the standard name in its preferred label field. In the NVS environment, " http://vocab.nerc.ac.uk/collection/P07/current/CFSN0023" is the unique URI (Uniform Resource Identifier) for the standard name "air_temperature" and it is equivalent to a GUID (Global Unique Identifier). The NVS guarantees its persistence based on the principles of "Cool URIs" (https://www.w3.org/Provider/Style/URI). Even if the domain name vocab.nerc.ac.uk needed to be withdrawn, we would arrange for the content of the NVS to move elsewhere and guarantee that the URI "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0023" will continue to resolve. Of course we would do the same for any other URL forms of CF standard names if this were required but the generation of the standard name URL (https://vocab.nerc.ac.uk/standard_name/air_temperature/) comes as an extra step, bespoke and specific to CF standard names and the P07 collection. Being an exception makes it more vulnerable.
This is why, when developing automated software or workflows, I would recommend building code based on the P07 URIs as primary sources of Standard Names instead of the standard name URL of the form https://vocab.nerc.ac.uk/standard_name/air_temperature/. But if humans are involved and preference is to use a non-opaque form of the URLs then "https://vocab.nerc.ac.uk/standard_name/air_temperature/" could be used (as Andrew said it is linked to the former using the owl:sameAs relationship).

[DocOtak]

I've done a first attempt at making the standard name xml from information contained only in the NVS. A commented on jupyter notebook is here: https://github.com/DocOtak/nvs-to-cf-xml/blob/master/nvs_to_xml.ipynb and the output is here: https://github.com/DocOtak/nvs-to-cf-xml/blob/master/nvs-to-std-names.xml

I found most of the implementation to be pretty straightforward but did find some issues that I think would need to be resolved or addressed before P07 is blessed.

Header info

NVS lacks a contact email, has a different institution, and does not have separate first_published and last_modified.

Standard Names

These were mostly straight forward to create, NVS lacks GRIB and AIMP mappings, so these were not included.

I found that 12 of the non-deprecated standard names have no unit information at all, I didn't check the deprecated names for units:

Concept ID	Standard Name
http://vocab.nerc.ac.uk/collection/P07/current/AKK6D0XA/	aerodynamic_resistance
http://vocab.nerc.ac.uk/collection/P07/current/EQUNJT0R/	isotope_ratio_of_18O_to_16O_in_sea_water_excluding_solutes_and_solids
http://vocab.nerc.ac.uk/collection/P07/current/CF12N559/	ocean_salt_x_transport
http://vocab.nerc.ac.uk/collection/P07/current/CF12N560/	ocean_salt_y_transport
http://vocab.nerc.ac.uk/collection/P07/current/OJUDV53W/	ratio_of_volume_extinction_coefficient_to_volume_backwards_scattering_coefficient_by_ranging_instrument_in_air_due_to_ambient_aerosol_particles
http://vocab.nerc.ac.uk/collection/P07/current/4FD2J2GJ/	storm_motion_speed
http://vocab.nerc.ac.uk/collection/P07/current/CF12N787/	tendency_of_sea_water_salinity
http://vocab.nerc.ac.uk/collection/P07/current/CFV8N121/	tendency_of_sea_water_salinity_due_to_advection
http://vocab.nerc.ac.uk/collection/P07/current/CFV8N123/	tendency_of_sea_water_salinity_due_to_horizontal_mixing
http://vocab.nerc.ac.uk/collection/P07/current/CFV8N124/	tendency_of_sea_water_salinity_due_to_sea_ice_thermodynamics
http://vocab.nerc.ac.uk/collection/P07/current/CFV8N125/	tendency_of_sea_water_salinity_due_to_vertical_mixing
http://vocab.nerc.ac.uk/collection/P07/current/S7ROPP0K/	volume_fraction_of_oxygen_in_sea_floor_sediment_pore_water

Aliases

I also found it somewhat difficult to map deprecated names to non deprecated names (aliasing), not all the deprecated names have a "replaced by" relationship or a "same as" relationship and I got caught by some cyclical relationships. I suspect "same as" will be better for our use cases since I believe that "replaced by" is a 1:1 relationship. The notebook has some more information on what I found difficult but I didn't fully explore this.

I think we do need a consistent way/mechanism that we will represent the aliases in NVS, this might just be clarification and cleanup on their side (Gwen?).

Units

I only used the content in P06 and found the altLabel to be pretty close to udunits already. There were some (15ish) exceptions that I manually mapped in the notebook. When investigating QUDT it appeared that many of the concepts lack the udunitsCode property, examples: J m-2, mol kg-1. We would need to contribute these mappings back to QUDT and probably check/verify the P06 to QUDT relationships are all in place too if we want to enable full programatic creation of the xml.

CC @gtesoro @japamment @ChrisBarker-NOAA

[larsbarring]

The difficulties you found regarding aliases seem (at least partly) to be the same that I found when working on cf-convention.github.io/#457. And I believe that the Vocabulary team will look at them one the CMIP7 standard name rush is over.

[taylor13]

I expressed some concern above about using the NERC server P07 as the reference source for standard name information. I now understand that the P07 NERC server provides useful information about standard names well beyond just recording the standard name and describing what it means, specifying its canonical units, and listing any aliases (and indicating whether it has been depricated). But I'm not convinced it is wise to consider the additional information provided by P07 of essential importance to the conventions. I would much prefer that we host on github a simple JSON file (or some similarly popular format) containing the essential standard name information. We would encourage others (e.g., NERC) to build on that the very valuable services they provide. We could steer users to the NERC services if they want to interact with other vocabularies without making their services essential for the existence of CF. I really appreciate that NERC supports CF, but can't we make the CF standard and any underlying databases independent of those who support it?

[graybeal]

Along the same lines as @taylor13's comment: I can't quite tell from context, but all the references to IRIs with P07 in the name have me worried. I distinguish where the standard names (meaning the local identifier parts) are served, from the persistent identifiers of the standard names. I think those two things are entirely separable.

WIth apologies if this is decided somewhere already, I hope the 'final' identifier of CF standard names (vocabulary and individual names) has a relatively short domain name+locator. Implementation-specific elements like P07 are undesirable in this context. I strongly prefer a vocabulary-specific IRI a la sweetontology.net, though something like http://vocab.nerc.ac.uk/ would be bearable if it's forever the authoritative location.

The key point being that where you serve the names from shouldn't affect their identifiers, which can be redirected to the actual service. This allows the service to change its DNS over time and insulates it from vagaries of the supporting organization.

[DocOtak]

Questions of stability of the NVS and specifically its domain name were asked at the CF workshop. @gwemon said that the NERC leadership has basically guaranteed that https://vocab.nerc.ac.uk/ would be the IRI even if NERC has name changes or other reorganizations. It has proven to be stable, maintained, and is critical infrastructure for SeaDataNet and the Argo Program. This is about as "forever" as I think we can get when it comes to computer and Internet infrastructure. Publishing on P07 is already part of the standard name process, and coupled with CF being an unfunded volunteer effort, I'd go with what already is already in place despite it maybe not being "pure".

[JonathanGregory]

Would it be useful to provide a URL for each standard name, like https://vocab.nerc.ac.uk/standard_name/air_temperature (which is an alias for the P07 page for that standard name - thanks, @DocOtak), but with only the CF information? In that page, we could give the link to https://vocab.nerc.ac.uk/standard_name/air_temperature for the sake of the extra information and facilities that offers.

If it is useful, I suppose those URLs could be provided on GitHub only as static pages - is that right? That might not be ideal.

If it were hosted somewhere else, could we create a subdomain of https://cfconventions.org to point to it? I don't remember who acquired and is paying for our domain! I hope that https://cfconventions.org will continue to exist for as long as the conventions are useful.

[ethanrd]

I believe CEDA owns the domain and manages the DNS records (I'm not finding a reference for that, so just my recollection).

Also, using the DNS records to point some aspect of cfconventions.org to the NVS came up previously (2020) in this comment from @graybeal in CF Vocab issue #11.

Are there any DNS experts in the house?

[DocOtak]

DNS can only resolve names and doesn't know anything about http. You cannot e.g. make vocab.cfconventions.org resolve to anything more specific than a domain like vocab.nerc.ac.uk using a CNAME record. So you cannot have https://vocab.cfconventions.org/sea_water_salinity resolve to https://vocab.nerc.ac.uk/collection/P07/current/CFSN0331/ either vocab.cfconventions.org needs to respond with a 301, 302, redirection, or vocab.nerc.ac.uk would need to know what to do with https://vocab.nerc.ac.uk/sea_water_salinity (in addition to the

On the CF side, we don't have our own servers, everything is coming off github pages. Which doesn't support the redirects we would want, you would need to do it with the html meta refresh tags.

Right now, cfconventions.org is using A records that point at github pages IP addresses.