CF Roadmap: Kicking off work on Provenance and Lineage theme

[erget]

Topic for discussion

Do you have ideas around data provenance and lineage, and would you like to be involved in ensuring that these themes are represented in a way that's beneficial to the CF Roadmap? This is the discussion for you.

At this year's CF Workshop we divided the roadmap preparations into multiple themes. This is one such theme. As you can see from the table, right now we're working on a green field with a blue sky - nothing really more than 2 keywords to start with. Behind that are a whole bunch of ideas and opportunites to shape where we go with it.

I'm setting up a call for later this month. If you want to be involved, please indicate your availability in this survey by the end of 14 Nov 2024.

Topics:

• what are our core concerns?
• who needs to be involved?
• what other technologies / projects should we be aware of and might we align with or adopt?
• anything else we should be aware of?

From there we can agree how to organise the work moving forward.

If you're not available on those dates but still want to be involved, that's no problem - let me know in the comments or drop me a mail and I'll keep you looped in :)

@pagecp

[JonathanGregory]

Does "provenance and lineage" mean discovery metadata?

[erget]

No, this is focused on being able to inspect data and find out how it was made.

[DocOtak]

Would this possibly be a revival of work on ACDD? I've also come across W3C's PROV a few times.

[erget]

I don't think it's got a good connect to ACDD, because it's not discovery but rather lineage. I like W3C's PROV a whole lot and agree that it seems like a good fit but we should analyse and verify.

[JonathanGregory]

Is it related to cell_methods, and the ways in which cell_methods needs to be enhanced, overhauled or replace e.g. to describe climatologies and anomalies wrt climatology? These matters are being discussed in Discussion 305 opened by @TomLav, following discuss issue 252, opened by @sethmcg, now superseded.

(The present discussion so far reads a bit like Twenty questions. 😃)

[sethmcg]

I think it is at least partially related, yes.

[larsbarring]

My understanding is that many of the mentioned issues and external resources have some overlap. I think that Daniel @erget captures it well:

... focused on being able to inspect data and find out how it was made.

At this general level the W3C's PROV that Andrew @DocOtak points at is the comprehensive resource. The abstract of the web page states:
"Provenance is information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness."

For users requiring this kind all-embracing machinery, maybe the best thing CF can do is to point at PROV in the conventions document, and by including a suitable string in theConventions global attribute.

Personally I think that the 2024 CF Workshop presentations by David Huard here, in the Uncertainty session, and by José Manuel Gutiérrez here, in the Statistical Processing session, offers a lot of food for thought.

Many users do not need e.g. manual quality judgement, expert group voting, and much more, but still do need something more than current cell_methods can offer. As Jonathan @JonathanGregory writes

... ways in which cell_methods needs to be enhanced, overhauled or replace ...

I think that by enhancing the cell method machinery we can cover several existing use cases, but I also think that we need to come up with something new that complements cell methods to cover even more complex use cases and data manipulation/processing. I have for some time been playing with the idea of creating some kind of "pseudo-language". That is, something similar to what is now used for parametric vertical coordinates but with freedom to also describe the equations. Well, still just some wild ideas without much substance behind ....

[erget]

Yes, in the end I reckon it's likely lthat PROV is the spec to use, not to re-invent - but it's still not clear to me how users should do that in a netCDF file, or if we think beyond netCDF in other CF-compliant data. So providing some guidance here so that there's 1 clear way to do it would probably be good. That's where I'm thinking the discussion will go - not necessarily rounding this out for before the roadmap exists but in the roadmap we can set forth what approach we want to use.

[larsbarring]

I fully agree! 👍 +1:

[sethmcg]

Over in discussion #305, we reached the conclusion that there are two levels of provenance to consider: there's the level that tells you how the data was changed such that you can understand what it means now, and there's the level that tracks what happened to the data such that you could completely regenerate it from the original sources by following the chain of provenance.

The first is what we're trying to figure out how we can accomplish with an overhaul to cell_methods. The second is like what gets recorded in the history attribute, but with maybe even more detail, and is (I believe) what we're contemplating in this discussion. The important distinction between the two is that the second is much, MUCH more complicated and detailed than the first, and involves not just abstract geophysical quantities, but entities like files and software and versions and configurations.

I agree that it's good to avoid re-inventing the wheel, and that it's probably best to make use of an existing spec like PROV, but if we're talking about recording a chain of provenance that is sufficiently detailed and specific to let you regenerate the end result completely, I would argue that it should not be mandatory, and definitely should NOT be incorporated into the netCDF file itself, because it can very rapidly become entirely unmanageable.

Recording every step in detail works fine if you're talking about a relatively short and linear chain of derivation, but if it's something more complex where it's branched or if you have a large number of input files, then things start to get out of hand really quickly. If you have multiple files as inputs, you have multiple history attributes (or the equivalent) to track / merge / reconcile / deal with. NCO takes the approach of adding them all to a history_of_appended_files attribute, and if you have more than just a couple input files, you end up with a gigantic header that consists almost entirely of history attributes.

This is not an idle concern; I've already run into problem cases where I had to turn off all history tracking while post-processing model output because I had files generated from hundreds if not thousands of parent files, and the resulting histories were preposterous. And that's just for doing data cleanup to make model output CF-compliant; the problem gets exponentially worse if you're doing things like bias-correcting data or calculating derived quantities with multiple inputs and intermediate outputs. (And if you really want to have nightmares, consider what happens if you do any of this in parallel by splitting files apart then recombining them...)

So I think this is an important concern for the roadmap discussion, and that we want to make sure we're not navigating toward the iceberg of metadata that's bigger than the data it describes.

[erget]

I'm totally on board with your concerns here - this is not something to be underestimated. I've been involved in some work in this area that I think might be useful for the community, because we tried to tackle many of the problems that you're referring to. I'm not claiming this is the only way to do it, or that the solutions we developed were bulletproof, but I think that they could be a useful starting point.

For the purpose of the roadmap the important questions in my mind are what we consider in the scope of our roadmap, i.e. what is important, where do we want to develop a stance, and what do we know about how that would look. And then some kind of horizon when we want to be there and whom we engage with.

More than willing to share the mysterious docs I'm referring to but I'd prefer to do that at a time when we meet - surely there'll be a time before end of Q1 2025 - so I can provide some context and we can discuss, otherwise it turns into a data dump.

But yes, my focus for this discussion is more on the history attribute rather than cell_methods, but I could imagine there are synergies there - I'm not an expert on cell_methods in any way!

[erget]

Hi folks, particularly @sethmcg and @pagecp - the survey has spoken, unfortunately there's no appointment for the near-term that fits for all of us but @sethmcg we'll fill you in and hopefully you can join later.

Anybdoy who wants on the invite let me know and I'll add you. @pagecp you're already on it. It'll be 2024-11-27T13:00Z on Teams.

[sethmcg]

@erget Go ahead and send me an invite. It's possible I'll be awake early enough to hop on.

[feggleton]

@erget do you have any notes from the meeting that you can share?

[erget]

Hi @feggleton , @sethmcg - sorry, I got a bit behind here in responding as I was out out office for a bit.

In the end it was just me in the meeting room so I figured I'd reach out again after New Year's when scheduling's a bit easier and give it another go. Hopefully participation will be easier then as well. I take it you're both keen to be involved? Then I'll reach out again and check your availability before setting a time.

[sethmcg]

@erget Yes, i'm keen to be involved. Thanks!