Profiles of use for CF roadmap publication

[lhmarsden]

Topic for discussion

CF profiles exist, for example WMO-CF, that reduce the degrees of freedom for encoding certain types of data.

It would be good if we provided a list of endorsed profiles on the website so data providers could find them more easily.

However, for the roadmap, do we only need to state our intent? We will not be creating profiles.

[taylor13]

I think if profiles are discussed, we should define what a profile is and indicate what (if any) criteria determine whether a profile will be endorsed. We should generically describe the kinds of restrictions/extensions that might be found in profiles, but not describe individual profiles in any detail. We can reference existing profile documentation for the interested readers.

[lhmarsden]

I agree with this

[lhmarsden]

In my opinion, a profile should only be endorsed if it is fully compliant with CF. I don't think we should endorse profiles that take only part of CF. For files types other than netCDF will this require some development of the conventions themselves?

[lhmarsden]

What if someone wanted to propose a new profile? Should there be a way for them to do this that involved CF reviewing the profile before including it on the website? Could this be done with github issues in the same way as the standard names are currently managed?

This could involve profiles not in NetCDF but in other data formats.

[jerabaul29]

Could there be a "RFC process" administered through a github repo, similar to what is used in, e.g., the Rust language? "It takes the time it takes", since a lot of it is open source, "on the free time" contributions, but the RFC system allows very good discussions and to develop very good and robust features :) .

https://rustc-dev-guide.rust-lang.org/walkthrough.html

[jerabaul29]

Would it make sense / be possible to add a global attribute profile="profile_name" to indicate what kind of profile is followed by a given dataset? That would make parsing much easier, rather than needing to infer with heuristics what kind of profile is followed? (@gauteh maybe you have thoughts on this too? :) )

[taylor13]

Could a profile be specified in the Conventions global attribute? I know multiple conventions can be included there.

[larsbarring]

I think (really guessing -- others will know more about this) that the concept of profiles was introduced within WMO activities. WMO lists three profiles here, of which two are now included in the Manual on Codes, Volume I.2 – International Codes. In this manual it is mandated that data adhering to either of these two profiles must include well specified strings in the global attribute Conventions. Some parts of these profiles may involve alternative mechanisms -- or even be inconsistent -- with the community version of the CF Conventions (i.e the one published on cfonventions.org).

I do not think that the CF Community has endorsed (whatever that might mean) these profiles, even though several individuals have made substantial contributions. I am not aware of other profiles than these three, but there might well be.

What might be the benefits if CF Community would begin endorse some kind of profiles?

[lhmarsden]

What might be the benefits if CF Community would begin endorse some kind of profiles?

For me the one benefit would be that it would increase the visibility of these profiles. This is somewhat related to this topic of having a catalogue of example datasets: https://github.com/orgs/cf-convention/discussions/355

For someone who is new to CF this would make it easier for them to know how to encode their (meta)data.

There are a lot of examples of where CF is being partially being used but not fully. I think this is clouding what is means for a dataset to become CF compliant. Listing what profiles are CF compliant is also intentionally not listing which profiles are not CF compliant. We could establish a dialogue between CF and those who are only partially using CF to advise on making these profiles fully CF compliant.

[ethanrd]

CF doesn't endorse software that uses CF or projects that use CF. Should CF be endorsing profiles other projects have developed or just listing and providing links to them?

[lhmarsden]

To me listing or providing links to projects/software/profiles is endorsing them. And in my opinion, CF should be doing this - though I suspect this could be a divisive topic.

[larsbarring]

Aha, ok, then I think that it is rather straightforward. Anyone can suggest updates to the two web pages Ethan mentions (and any other pages, too, for that matter). For the software page this has happened several times, which always have been well received when the proposer is able to suggest text. For the projects page, that I compiled just to get the ball rolling, additions have not yet been suggested. Additions are most welcome!

[lhmarsden]

Ah! I didn't realise that profiles were already listed there.

[feggleton]

Yes I think listing and linking to them is great to do but not specifically stating they are endorsed.

[jerabaul29]

In addition to what is being described here, as a practitioner, one thing that would be very useful is short, straight to the point "minimal complexity examples" of how to do things right. For example it would be very useful for each profile / main "dataset structure" to have a short text description (at most 1 A4 page) + e.g. python / xarray script (possibly leveraging existing "data packing helper packages", for example in the domain I am interested in i.e. buoys data we are working with some colleagues on https://github.com/OpenDrift/trajan ) + sample output .nc file generated from the script. Each of these could be hosted on github in a repository rather than on a static website, so that people could interact / discuss etc through the issue tracking system, so that there can be a 2-ways dialogue and discussion about each profile.

[lhmarsden]

We had a discussion in the roadmap meeting

Some points were discussed.

• We should find out what profiles are out there - though they might not be included in the roadmap paper.
• To what extent are they compliant?
• The criteria for what a profile should comply with could be outlined in the roadmap paper.
• Do we want to list some profiles in the roadmap paper?
• Could there be mechanism to help profiles that are close to fully CF compliant to become fully compliant?
• Implications of listing a profile (or software) on the website that is not fully compliant?

My personal opinion is that the roadmap should focus only on outlining what criteria a profile should adhere to. Perhaps a definition and/or a checklist. Maybe a sentence saying that whilst CF will not actively be taking on the task of 'creating' profiles, it encourages any activities that reduce the degrees of freedom between similar datasets.

@taylor13 mentioned that one obvious criterion is that a profile should pass the CF checker and I agree with this. A NetCDF file does not have to contain a lot to pass the checker, but as discussed elsewhere, developments to the checker will help with this and are desired/planned.

Does anyone have any sugestions for additional criteria?

[taylor13]

Of course, I'm not sure how'd we'd enforce the "checker" criteria I suggested since some but not all files produced under a profile might conform.

[lhmarsden]

I think if a profile is well defined and fully CF compliant and reduces the degrees of freedom then all datasets that conform with the profile should also conform with CF.

In my opinion we don't need to enforce any criteria, just state what the desired criteria should be. I think this would help people to design profiles and improve those that are not fully compliant.

[taylor13]

O.K. that seems like a practical way forward, and I agree then that "enforcement" should be outside our focus.