Incorporating the CF data model into the conventions

[davidhassell]

Title: Incorporating the CF data model into the conventions
Moderator: @JonathanGregory
Requirement Summary: A data model allows for a more structured development of the CF conventions.
Technical Proposal Summary: A new document describing the CF data model is incorporated into CF, and new rules for keeping it up to date.
Benefits: Provides structure to proposers of enhancements to CF, and creators of CF-compliant software
Status Quo: CF continues to not benefit from having a formal data model.

Detailed Proposal:
In this ticket we propose that the CF data model for data and metadata corresponding to the existing standard (version 1.7) is incorporated into the CF conventions.

This follows many years of on-line discussion, the external publication of the data model (GMD, https://doi.org/10.5194/gmd-10-4619-2017) and the consensus to incorporate the data model that was reached at the 2018 CF-netCDF meeting in Reading.

The full description of the data model, as it would be included in CF, may be found at https://github.com/davidhassell/cf-data-model/blob/master/cf-data-model.adoc. This document is almost entirely a cut down version of the GMD paper.

It is proposed that this description of the data model resides in a new repository called https://github.com/cf-convention/cf-data-model (which does not yet exist).

The CF data model document contains background information on what the data model aims to achieve, its design criteria, and relationship to the current netCDF encoding.

Note that the data model published in GMD was for CF-1.6, but it is consistent with all of the changes to the conventions arising from the 28 tickets that contributed to CF-1.7. Therefore we may consider this also to be
the data model for CF version 1.7.

Governance:
The CF data model will guide the development of CF by providing a framework for ensuring that proposed changes fit into CF in a logical way, rather than just a pragmatic one.

To ensure this occurs, some additional rules for conventions changes have been drafted. These are included in pull request cf-convention/cf-convention.github.io#72

In brief, a proposed change should be assessed for data model compatibility. If it is not compatible, and the proposal cannot be acceptably modified so that it is, then the data model will be changed to accommodate the change. The proposer is not required to have detailed data model knowledge - the new rules state that "The assessment will be carried out by a member of the conventions committee or another suitably qualified person. If no-one volunteers, the chairman of the committee will ask someone to do it".

[erget]

@davidhassell I haven't been able to find the associated PR for this change - are you sure you made one?

I have glanced over the file you propose adding to the Conventions but have not had time to review it in detail. I have a high degree of trust in it because I remember the paper it's made from and the discussion we had surrounding it, but in the interests of transparency I have not done my due diligence on this proposal yet.

At this stage I can definitely say that I like the approach - adding the model to the conventions will make things much easier for people implementing software to understand how CF works - and indeed if they're not computer science people I think it provides a lot of value nonetheless!

I'm not sure why you propose housing the data model itself in a separate repository - in general I would advocate keeping it in the current one, could you explain the motivation for separating it? Is it because you want to have a clear versioning of the data model that would make it obvious when it changes, across different versions of the Conventions? I'd be interested in the motivation for the second repository.

Have a nice weekend!

[davidhassell]

Created a cleaner Pull Request for the changes to the governance rules: pull request cf-convention/cf-convention.github.io#72

[davidhassell]

@erget Thanks for your support, and transparency - it would be very useful if you, and others, could find the time for some due diligence, at least to check that nothing got missed out in the cut-and-paste process.

At present there isn't a PR, as I was proposing a separate repository. I must admit that I am a little fuzzy as to why I suggested a separate document. I thought that was what was suggested at the 2018 CF-netCDF meeting in Reading, but going back to my notes, they just say "separate document or appendix?" If I could start over, I would prefer an appendix.

This gets me thinking that perhaps the conformance document should be an appendix also? This gets round having to do issues and PRs in multiple repositories when making changes. If everything was in seperate repos, a change that needs a data model update and conformance rules changes would need 3 PRs!

[erget]

I agree, and additionally it opens up the possibility for things to get implemented without having applied the full scope of necessary changes. If we combine all sources into one repository we can put a checklist into place, so that even if we don't need to update the data model and conformance document, at least we've checked to verify that no update is needed.

[JonathanGregory]

As moderator, I propose that this issue be agreed with the change discussed above, that the data model document should be in the same repository as the conventions document. I believe that Daniel @erget and @davidhassell are now of that opinion too. Please say if that's not right.

My view is that it should not be an appendix to the CF conventions document, because it's not part of the netCDF convention; I would say that it's even less part of the convention than the conformance document, since the data model does not refer to netCDF. Does anyone want to argue that it should be incorporated on other grounds?

Jonathan

[erget]

For what it's worth I agree with @davidhassell that having the data model as an appendix to the Conventions, for the reasons he lists above. However I would prefer having the document in the repository than elsewhere if it is not in the conventions document. Its independence from netCDF is in my opinion an advantage, rather than a disadvantage, and would be helpful to those who are not familiar with the format but have to work with logical representations of data encoded in it.

[davidhassell]

I have moved the CF data model description to a new appendix, I, as discussed above. See the following pull request.

The PR also introduces some images in an images folder - I am presuming, for now, that this structure is OK for rendering the content correctly via the website.

The data model at this time is still at CF-1.7. I will post the changes required for CF-1.8 initially as comments in this discussion, prior to incorporating them into the PR.

[JonathanGregory]

As far as I know there aren't any disagreements on this issue and the pull request as described will be acceptable to all. If anyone disagrees, please say so (within three weeks). Thanks. Jonathan

[JonathanGregory]

Since no-one has objected and enough support has been expressed I conclude that we have agreed on this issue. With the data model forming a new appendix, David should join the list of additional authors of the conventions document. Thank you, David.

[davidhassell]

Thank you, Jonathan, and everyone over the years who has contributed to getting us this far. The new content will be merged soon.

[davidhassell]

I have just noticed that not everyone here is also watching the repository that serves the CF conventions web site (which is absolutely fine!), but there is an issue there that is intrinsically linked to this one: cf-convention/cf-convention.github.io#71 (Rules for updating the CF data model). It describes how the data model should evolve itself, and influence the evolution of the CF conventions. Any comments on that issue would also be gratefully received.

Thanks,
David

[JonathanGregory]

Implemented by cf-convention/cf-convention.github.io#72