Can CF offer better support for the representation of climate anomalies?

[TomLav]

Topic for discussion

We continue the discussion started as issue #252. [It's now vocabularies issue 70, but you can still reference it as discuss issue 252.]

A summary of the discussion so far is in this post.

Tagging @JonathanGregory, @davidhassell and @sethmcg.

[Discussion #372 may be relevant. It was opened by @larsbarring in 2019 as issue #197 in the conventions repository, and I have converted it to a Discussion. Jonathan]

[sethmcg]

Thanks for the great summary and for starting this discussion, @TomLav!

In response to your questions:

Do we have a shared understanding of this thread so far (my partial summary)?
Do we agree that we have identified a shortcoming in the way CF currently handles anomalies and their link to climatologies?
Would it be of interest to try and propose a solution for that particular issue?
Would it be best to continue the discussion in this thread, or start in another one?

I think your summary is an excellent starting point, I agree that we have identified a shortcoming, and I think that we should try to propose a solution for it here in this discussion.

@sethmcg do you want to move your questions forward in parallel? Or do we first try and "fix" the first issue at hand?

I would like to move those questions forward as part of this discussion, because I fear that if we don't keep them in mind in trying to address the first issue, we may inadvertently cut ourselves off from a solution to the full set of issues.

So, to summarize my last two posts: currently, the way we use cell_methods and climatology records the periodicity and coverage of the data you started with and how you aggregated it, but it doesn't say what the periodicity and coverage of the result is; that's recorded implicitly in the values of the time coordinate. This creates ambiguity and makes it hard to interpret the results if you aggregate more than once. This should be easy to fix by also explicitly recording what the result was, instead of just leaving it implicit. However, is there one more ambiguity, which is that if you have both cell_methods and climatology, you don't know which order they were applied in. So we need to record that, too.

(And once we start considering how to do that, it starts to look like maybe we shouldn't be thinking of climatology as something different and separate from a cell_method; they're both just aggregations.)

[TomLav]

@davidhassell : you mentioned (here) you had some comments you would to make on this topic?

[TomLav]

Hello. This discussion has not progressed, probably by lack of time before the summer and the summer break.

From experience with past CF discussions, what is the best way to move such things forward? Ultimately we want to propose improvements to the CF convention (a new section?) to help better describe anomalies. Should this be done through online meetings? I am tagging here the people who showed interest so far: @davidhassell @sethmcg @JonathanGregory

[sethmcg]

My suggestion would be to float an idea. People often need something to react to to get things going.

So: if we were starting from scratch, I think the way to do this would be to have an attribute named derivation that listed, in order, the transformations that were applied to the data. So you could have something like:
tmax80reganom@derivation = "interpolate from native grid to 0.25 degree; daily maximum from hourly; monthly average from daily; 80th percentile over region; monthly climatology from monthly across years 1980-2010; monthly anomaly from monthly climatology"

That's an extreme example, but would something like that capture all the information we want? How would it relate to @cell_methods and @climatology? Is it extensible? Does it invalidate old data? What do people think, is this the general direction to try to go in?

[sethmcg]

A quick addendum to this suggestion, prompted by discussion in #341:

It would probably make things significantly more human-interpretable if, rather than using a single string with a delimiter as in the example I provided, we used a string array to record a sequence of transformations in a derivation attribute. I.e.:

tmax80reganom@derivation = "interpolate from native grid to 0.25 degree",  
                           "daily maximum from hourly",
                           "monthly average from daily",  
                           "80th percentile over region",  
                           "monthly climatology from monthly across years 1980-2010",  
                           "monthly anomaly from monthly climatology"

So assuming that we don't end up with a total ban on string arrays, I think we should consider that as a possible element of the solution.

[TomLav]

Thanks for moving this forward. I can see that derivation gives a useful human-readable description of what was done with the data.

But what about the machine-readability? Would you aim at machine readability (parsing, grammar, etc...) as well ?

Also, if a variable holds two climatology maps (dims: lat, lon, nb_clim=2): one against 1981-2010 and the other against 1991-2020). Then I am not sure how to adapt the derivation idea.

[sethmcg]

What do we mean by machine-readability? It would be nice to have, but I'm not sure it's possible. For example, one of the transformations we might want to record is bias-correcting the data, which can be a very complex operation like using the MBCn algorithm (which does multi-variable trend-preserving quantile mapping in an iterative loop that randomly rotates and un-rotates the variables before and after). What does it mean to record that operation in a machine-readable way?

Good point about climatology maps over different periods. Definitely something we need to discuss.

I think what's happening in general terms is that we're talking about a variable that has a dimension that corresponds to variations on the chain of derivation. It would be similar to a variable that had a quantile dimension. One of the issues that led to this discussion is the fact that the existing mechanisms don't fully capture the way that the transformations affect the dimensions of the data.

So I think that means we need to flesh out the idea I've floated a bit more. Currently it describes what the transformations do in human-readable terms, but it doesn't formally record the changes to dimensionality. That example is pseudo-code, basically. If we developed a notation for how the dimensions change (time dimension at daily frequency => climatological time dimension over different periods), do you think that would cover it?

[larsbarring]

Hello @TomLav
Thanks for resurrecting this issue, and thanks @sethmcg for initial suggestion. This is very timely and highly relevant in relation to the CF2024 workshop in less than a month. In fact there is a specific session where this topic fits perfectly:

Tuesday 17 September 16:00 - 17:30 CEST (UTC+2): Metadata for describing statistical processing

It would be excellent if you were able to join this session (as well as the whole workshop of course). Moreover, if you have more concrete ideas that you would like to work on hackathon-style (maybe as an outcome of the conversation during the session) do not hesitate to suggest a hackathon theme in this discussion topic.

[sethmcg]

I can't make the rest of the workshop, but I plan to (virtually) attend that session!

[larsbarring]

Great! Just friendly nudge to register here even if you plan to attend virtually for just one session. This will help us allocate "connection space" or whatever our IT/web support call it.

Thanks /Lars

[sethmcg]

Already done!

[DanHollis]

I have some interest in this topic. From scanning the discussions so far it seems like this could get pretty complicated! My initial thoughts are that there are three things we need to capture:

• The aggregation process that was used to compute the variable (captured by the existing cell_methods, I think)
• The aggregation process that was used to compute the climatology (this could be captured by a second 'cell_methods' that is part of a dummy climatology variable)
• The method by which the data variable and the climatology are combined to obtain the anomaly e.g. difference or ratio (this could be captured by the suggested derivation attribute)

[DanHollis]

Here's a concrete (and fairly simple) use case, in case it helps with the discussion.

I work with gridded datasets derived from in situ station observations. One example of a parameter we routinely generate is the monthly mean of daily maximum temperature. Our grids for individual months (e.g. July 2024) have cell_methods="time: maximum within days time: mean over days". The corresponding climatology (in our case, the 1991-2020 climate normal for July) has cell_methods="time: maximum within days time: mean over days time: mean over years". For the anomaly we calculate the difference between the individual month and the climate normal. For other variables, e.g. precipitation, we express the monthly value as a percentage of the climate normal.

A couple of things occured to me while writing this:

The fact that these data are computed from daily maximum temperatures is obviously a key part of the description. However it is not actually a processing step that data providers such as myself would ever undertake. It is either a characteristic of the observation itself (in the case of liquid-in-glass thermometers) or it is a function of the data logger (in the case of electrical resistance thermometers).

Secondly, in our case both the gridding process and the calculation of the anomaly come after the aggregation steps described in the cell methods have been applied to the station data. However, other approaches are equally valid e.g. creating a separate grid for each day from the station maxima and then creating the monthly grid by averaging the daily grids. In this case 'gridding' comes between time: maximum within days and time: mean over days.

[larsbarring]

For the use case of maximum temperature anomaly, what standard name are you using for the anomaly (there is the air_temperature_anomaly that I think is applicable)? In in relation to this there is an open issue cf-convention/vocabularies#27 on specifying the reference_epochfor anomaly terms. But this, I gather, only covers the situation that the same dataset is used to calculate the reference climatology.

[DanHollis]

To be honest we have been fairly lazy in this regard. All of the data we save to disk are actuals rather than anomalies, so we don't have an issue there. The main time that we need grids of anomalies is when we are generating maps - this is done on-the-fly i.e. the output is an image, not a netCDF file. To achieve this we add our own bespoke attribute to the data in memory to label it as anomalies (the standard name does not change).

Clearly we could use air_temperature_anomaly. However, we process quite a few other variables as well (e.g. precipitation amount, sunshine duration, frost days, rainfall days, wind speed) and, as this thread is highlighting, we'd need more than new standard names to fully describe the data. I haven't had time to pursue this, plus I didn't really have a strong requirement (because we are not saving the anomalies to disk), so we've just improvised our own solution sufficient for our needs.

Having said all that, some months ago we did have a request from an internal user for netCDF files of anomaly values. If this discussion leads to a way to make these files CF compliant then that will be great from our point of view.

One thought regarding machine-readability - We use our bespoke attribute to distinguish actuals from anomalies and, therefore, to create appropriate titles and legends for our maps. I guess new standard names will be required anyway, however I wouldn't want to be parsing the standard name to work out if the file contained anomalies. If the eventual solution can provide a machine-readable way to identify the file contents as anomalies then that would allow us to drop our bespoke attribute.

[sethmcg]

@DanHollis -- Thanks for those comments! Those are very helpful examples and really clarified for me what it is we are trying to accomplish here.

The reason why we need to record the sequence of transformation applied to a variable is not to record the provenance in a way that lets you perfectly recreate the workflow that generated the file, but to record the chain of derivation that changes the meaning of the data.

So the point is to have something that will let you clearly distinguish between things like "climatological mean of the monthly maximum of tmin" and "climatological minimum of the monthly mean of tmax" that can be confusing or ambiguous under the current representation scheme that uses only standard_name, cell_bounds, and climatology attributes.

That's a very helpful reduction in scope, because it dramatically simplifies what we need to record, and I think makes it far more achievable to come up with something that is both human- and machine-readable. For something like a bias-correction transformation, we don't need to record all the details that would let you reproduce it, just that bias correction has happened. Ancillary information like (type of) algorithm, observational dataset, and reference period are important, but we don't need things like implementation details, library versions, parameters, etc.

Great! I think that's doable.

[DanHollis]

...to record the chain of derivation that changes the meaning of the data.

Yes, definitely! In the example I gave the resultant quantity is actually the same (monthly mean of daily maxima) however changing the order of operation ('averaging then gridding' vs 'gridding then averaging') will inevitably produce a different result. In your example the resultant quantities are different things, so to describe them unambiguously is essential. I don't know if this distinction is important but I thought I'd mention it - I imagine any flexible solution will handle both scenarios.

I agree that the aim should not be complete reproducibility. Instead we are just helping data users to understand whether they are comparing apples with apples or apples with pears (which fundamentally is what CF is all about).

[JonathanGregory]

Dear all

Unfortunately I haven't had time to think about this issue in the months since it began. From reading through it just now, I agree with the scope described by @sethmcg and supported by @DanHollis: The reason why we need to record the sequence of transformations applied to a variable is not to record the provenance in a way that lets you perfectly recreate the workflow that generated the file, but to record the chain of derivation that changes the meaning of the data.

But this is the aim of cell_methods. Therefore I wonder whether we can add more kinds of operation to cell_methods to meet the needs discussed here. Calculating a climatology, calculating anomalies wrt the climatology, and changing the spatiotemporal resolution of the data strike me as operations that could be described as cell methods.

Unfortunately in Sect 7.3 we don't state what the purpose of cell methods is! Essentially, it is to record how the value in the cell relates to the variation of data from which it was computed at a different resolution and over other dimensions. As in this discussion, an important feature of cell_methods is that it's an ordered list of operations. In the paper about the CF data model, sect 3.6 discusses it, and it's summarised by the data model in Appendix I as describing variation within cells.

Best wishes

Jonathan

[sethmcg]

Hi Jonathan,

I think we could do that, but in order to cover everything that needs to be covered, we'd have to change what cell_methods means. I'm not opposed to that! But it will be a pretty significant overhaul.

Appendix I says that cell_methods describe "how the cell values represent the variation of the physical quantity within its cells" (emphasis added). We would need to change that to a broader definition like the one you suggest. I think we'd need to simplify it and just say "how the value in the cell relates to the data from which it was computed" to cover everything we need to. (Climate anomalies, for example, don't change resolution or depend on other dimensions, they're just a value relative to a reference period, which can lie entirely outside the temporal cell of the anomaly.)

Given that change in the definition, we can add more kinds of operations, but I think there's one additional piece we need: cell_methods needs to be explicit about how the operation changed the data. Currently it's implicitly "from whatever it was originally to what it is now", but that's not enough if you have more than one operation affecting the same axis. (Not to mention that describing "what it was originally" is optional, and "what it is now" must be inferred from coordinates and/or other attributes.)
So we need to expand the cell_methods syntax to include (mandatory) before and after information.

(Actually, I think there's a second piece we need: we also need a delimiter between operations. If it's becoming more complex than just axis: statistic, I don't think a space-delimited string is parseable anymore.)

Does that seem sensible to other folks? Is this the overall direction to head in?

[TomLav]

Does that seem sensible to other folks? Is this the overall direction to head in?

Since you are asking, I am just going to chip in and mention that I had opened this discussion with a very specific need in mind: to improve the description of anomaly products in CF. CF doesn't do this today, it has an half-finished, half-documented solution (using reference_epoch) that producers (at least me) have to guess how to use properly. Then we found that this half-finished solution will cover many cases, but not 100% of them. There was thus already a proposal by @JonathanGregory on how to move a step forward (using a placeholder climatology variable, with the existing cell_methods).

All this to remind us that there is a pressing need today around anomalies in CF. Maybe I am missing us sharing a good understanding of what the current (half-finished) solution is, then how a simple (?) step like the placeholder climatology variable would improve it, and finally how cell_methods syntax should be revised. Maybe I am wrong and the revision will solve everything in the long run. Maybe I am having cold feet. And unfortunately I have too little time to offer this community on this topic at the moment.

[JonathanGregory]

Dear @TomLav

I hear your pressing need for a solution to a specific problem. Please could you remind me (us) of the use-cases which you need to be able to handle now, for which the current convention is insufficient? Maybe we can devise something limited and adequate.

Dear @sethmcg

I would be willing to join in with reviewing, and if necessary expanding, generalising or overhauling cell_methods if it is not sufficient for use-cases that you or others have. There are two open conventions issues about cell_methods as well (#414 and #447). Since it's complicated, I expect it would take considerable time.

If it turns out that cell_methods cannot be used or modified to do all that is needed, I would advocate replacing it with a new attribute, and supporting both in future, only one of which could be used for any given data variable. The new attribute would take on all the functions of cell_methods. I think that would be better than providing another attribute that somehow had to be used in combination with cell_methods. Since the functions would probably similar, that would be confusing. However, an entirely new attribute would be a larger requirement on software developers and an obstacle to useability of data, so I hope that we can avoid it.

Best wishes

Jonathan

[sethmcg]

That makes sense to me.

[TomLav]

Dear @JonathanGregory,

Sorry for the delay in returning to this.

I hear your pressing need for a solution to a specific problem. Please could you remind me (us) of the use-cases which you need to be able to handle now, for which the current convention is insufficient? Maybe we can devise something limited and adequate.

I do not think my use cases are particularly difficult. I typically want to store an anomaly map as a monthly mean (say mean Feb 2024) minus a monthly mean map over a set of years (say mean of Febs over 1980-2010).

I believe this is covered by the CF convention using the reference_period mechanism.

However, as I am alluding to here I think CF currently lacks documentation about how to use reference_period for anomalies. We have a whole chapter on Climatological Statistics but nothing on Anomalies. Improving the documentation of how to store anomalies (using the reference_period mechanism) in the CF document would be already very helpful.

Thomas