CF adoption by large projects?

[taylor13]

Question

Does anyone know the extent to which CF has been adopted by large projects? I am well aware of its use in most "model intercomparison projects" like CMIP and in surrounding projects like obs4MIPs and input4MIPs. What I'm wondering is outside the MIPs, has CF been adopted? For example, any of the reanalyses (ERA, for example) or Copernicus. I know ECMWF only really supports grib, but they offer some datasets as netCDF files, and it's not clear to me whether they are CF compliant.

If you know of any large-scale adoption of CF outside the MIPs, it would be nice to perhaps list these somewhere on our website.
thanks,
Karl

[JonathanGregory]

Dear Karl @taylor

I asked ChatGPT, "Do you know of any large projects (excluding CMIP) that have adopted the CF metadata convention for their netCDF files?" and it replied (rather slowly and hesitantly, compared with its usual speed):

Yes, several large projects and data repositories have adopted the CF (Climate and Forecast) metadata conventions for their netCDF (Network Common Data Form) files. Some notable examples include:

• European Space Agency Climate Change Initiative (ESA CCI): The ESA CCI project uses the CF conventions to ensure consistency and interoperability of climate data across various Essential Climate Variables (ECVs).

• Earth System Grid Federation (ESGF): ESGF is a collaboration involving multiple international organizations that provides access to a vast array of climate data. The CF conventions are widely adopted within the ESGF framework to ensure standardization.

• National Oceanic and Atmospheric Administration (NOAA): NOAA's various data repositories, such as the National Centers for Environmental Information (NCEI), use CF conventions for many of their climate and oceanographic datasets.

• Copernicus Climate Change Service (C3S): Part of the European Union’s Copernicus Program, C3S provides comprehensive climate data and services. It employs CF metadata conventions for consistency and to facilitate data integration.

• Joint Effort for Data assimilation Integration (JEDI): This project, led by the Joint Center for Satellite Data Assimilation (JCSDA), uses CF conventions to handle diverse data types in a unified framework for weather prediction and climate research.

• Arctic Data Center: Managed by the National Center for Ecological Analysis and Synthesis (NCEAS), this repository adheres to CF conventions for its extensive collection of Arctic research data.

• Australian Integrated Marine Observing System (IMOS): IMOS provides oceanographic data following the CF conventions, ensuring that their datasets are accessible and usable by the wider scientific community.

These projects and repositories use CF conventions to enhance the interoperability, accessibility, and usability of their climate and environmental data.

In the small print at the bottom of the page, it notes "ChatGPT can make mistakes. Check important info."

Best wishes

Jonathan

[sadielbartholomew]

+1 for a clever use of ChatGPT! That gives us a good initial list after some fact-checking 🙂

[taylor13]

thanks, @JonathanGregory, to you and your assistant. This is very helpful. Can anyone vouch for what these two have uncovered, and perhaps expand on the extent to which the data holdings of these projects conform with CF?

[erget]

Hi folks,

I can corroborate a bit, and also add all of EUMETSAT's next-gen satellites. EUMETSAT provides 45% of the climate data records used in the IPCC reports, see e.g.
https://www-cdn.eumetsat.int/files/2023-06/AR35_EUMETSATAnnualReport2022_web.pdf page 34:

I believe that's pretty significant use of the CF Conventions!! We've got Meteosat Third Generation launched and almost ready to go (geostationary), EPS Second Generation around the corner (big flagship polar orbiters), plus numerous missions for the European Union and partners that we're contributing to and all of which will be using CF.

We might also add the modelling outputs of the European Centre for mid-range weather forecasting - as I understand it they have expanded their use of CF significantly over the past few years.

Group for High REsolution Sea Surface Temperature (GHRSST) also comes to mind, as does the WMO, who have published WMO CF Extensions to encourage the use of CF by the national weather centres around the world.

In terms of corroborating point by point, without consulting ChatGPT:

• ESA CCI - yes, they're heavy users
• ESGF: Here I only know rumours, not familiar with their work
• NOAA: NCEI is definitely CF-based, or at least it was when I was seconded there for a summer. I would be extremely surprised if that had changed.
• C3S: Also makes big use of CF
• JEDI, ADC, IMOS: I am not familiar with these.

[larsbarring]

We should definetely include CORDEX (www.cordex.org), which probably goes under the umbrella of "MIPs" (even though that is not in the acronym as such.) But it draws heavily on CMIP for general organisation of data request, variable naming, attribute conventions, and more. And all of it should (must! :-) be CF compliant.

[ChrisBarker-NOAA]

as for :

National Oceanic and Atmospheric Administration (NOAA): NOAA's various data repositories, such as the National Centers for Environmental Information (NCEI), use CF conventions for many of their climate and oceanographic datasets.

I would say that's more aspirational than enforced. I don't know what NCEI's policies are, but at least the NOAA Ocean Service oceanographic models do not put out CF compliant results :-( -- I am working on that, so hopefully one day, but not today.

[taylor13]

thanks. That's helpful.

[sadielbartholomew]

Great idea, Karl. It would be great to advertise the wide and international adoption of CF.

Don't forget The National Aeronautics and Space Administration (obviously a particularly well-known organisation to even those outside the field), at least assuming the notes I see on the web are correct to some extent and that we can include in the cases where sub-projects have adopted, with that noted clearly. For example one project (ESDIS) states from the 'Standards and Practices' page here that:

Status
The Climate and Forecast (CF) Metadata Conventions is an approved standard recommended for use in NASA Earth Science Data Systems in April 2010. At that time, the current version of CF was 1.4. The CF conventions are regularly updated by a robust and well-documented community consensus process. Thus, ESCO recommends using the most current, released version.

though I don't know the full extent of NASA's adoption or otherwise of CF for their earth (or otherwise) data. Can anyone provide further context on the topic, and/or confirm or otherwise if we can include NASA and/or a sub-project/domain of it, since I think it would be an impressive addition to the list if we can justify adding it?

[larsbarring]

Some time ago there was a similar issue. I think that it would be very useful to have such a list in place on the website before the CF2024 workshop in September. I will try to put up an initial PR linked to cf-convention/cf-convention.github.io#256. Of course the list will grow with time.

[larsbarring]

There is now a draft pull request for adding such a table to the website. Please help to improve this draft by commenting in the associated issue cf-convention/cf-convention.github.io#256, or in the PR if it is of technical nature.

[larsbarring]

The draft pull request was merged some time ago. But as I noted in cf-convention/cf-convention.github.io#256
this is just a first version based on the limited information available to me at that time. There are huge gaps in coverage -- please help to fill those!

[taylor13]

Where can I view the current version suggest additions?

[JonathanGregory]

I believe it's the page on Projects and Activities that Use the CF Metadata Conventions on the CF home page.