POC: Handle std names and aliases (#5257) #5313

larsbarring · 2023-05-11T10:05:30Z

🚀 Pull Request

Description

This is a "proof-of-concept" PR to address how to better handle standard name aliases. It consists of the following elements:

Reorganising the iris.std_names.py a bit to have a separate dict for aliases (via updated tools/generate_std_names.py).
It now includes some table version information (mentioned in CF global attribute "Conventions": expose to users, and include CF standard name table version #5255), and a separate dict for the standard name descriptions (optional when generated).
Adding a new std_name_table.py containing the following functions:
- get_convention -- return a tentative Conventions string
- set_alias_processing -- define how to handle aliases:
  "keep" - current behaviour, treat aliases in the same way as currently valid standard names,
  "warn" - issue a warning (default), otherwise as "keep",
  "replace" - silently update aliases to current standard names.
- get_description -- return the standard name description if available
- check_valid_standard_name -- check if a name is a standard name or an alias, and do the translation if requested as defined by set_alias_processing
std_name_table is [naively] imported in iris.__init__
common/mixin._get_valid_standard_name is modified to use check_valid_std_name

No units test have been added (would be good to first get some feedback whether this POC is a reasonable approach .... )

Consult Iris pull request check list

codecov · 2023-05-11T10:18:52Z

Codecov Report

Attention: Patch coverage is 45.00000% with 33 lines in your changes are missing coverage. Please review.

Project coverage is 89.19%. Comparing base (a3931f6) to head (d2216e7).
Report is 300 commits behind head on main.

Files	Patch %	Lines
lib/iris/std_name_table.py	35.29%	30 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5313      +/-   ##
==========================================
- Coverage   89.31%   89.19%   -0.13%     
==========================================
  Files          89       90       +1     
  Lines       22375    22430      +55     
  Branches     5368     5383      +15     
==========================================
+ Hits        19985    20007      +22     
- Misses       1640     1670      +30     
- Partials      750      753       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

https://github.com/cf-convention/discuss/issues/229 Standard names: hard to spot "typo" in `surface_upward_mass_flux_of_methane_due_to_emission_from_fires`

* new optional argument -d/--descr for including standard name descriptions * including the description of all standard names * including standard name table version and related info. * now contains several variables: - variables (starting with underscore) used in the processing - VERSION (dict) - CONVENTIONS_STRING (str) - STD_NAME (dict) - ALIASES (dict) - optionally DESCRIPTION (dict) harmonize quotation marks to single quotes in description string

CLAassistant · 2024-04-08T16:15:03Z

All committers have signed the CLA.

pp-mo · 2024-05-01T10:00:36Z

@SciTools/peloton is this still alive issue, and if so could you sign the CLA @larsbarring ?

larsbarring · 2024-05-01T10:12:42Z

I thought I had already signed the CLA because of one or two previous minor contributions. Anyway, now done. Whether it is alive or not I am not sure. I have not made any further effort since back then as I am not sure whether it is a reasonable approach in the context of Iris.

pp-mo · 2024-05-01T14:47:58Z

I thought I had already signed the CLA because of one or two previous minor contributions. Anyway, now done. Whether it is alive or not I am not sure. I have not made any further effort since back then ...

Ok thanks!

I am not sure whether it is a reasonable approach in the context of Iris.

Well, I guess we'll take a look + see about it
@larsbarring are you at least clear that something like this would still be useful ?

larsbarring · 2024-05-01T16:59:19Z

Yes, I think that this would still be useful. In the context of CF an aliased standard name is typically regarded as deprecated. Software should be able to read data having an aliased standard name, but new data should use the replacement name. Obviously, there are judgements to be made here regarding how to deal with this in practice. But and aliased standard name should not be considered just as an alternative at the same level as standard name.

Also, note that there will likely be some minor changes to the standard name xml file format as of next version, possibly also backported to all previous versions.

schlunma · 2024-05-02T15:16:18Z

A proper way to handle standard name aliases would also be useful for ESMValTool, see ESMValGroup/ESMValCore#1985.

One issue we currently face is merging cubes with different standard names, which is (in the current iris version) not even allowed if the standard names are aliases of each other.

bouweandela · 2024-06-06T09:09:38Z

@LisaBock this may be of interest to you.

HGWright · 2024-06-12T09:30:27Z

@SciTools/peloton We are planning to talk about this on the 3rd July.

trexfeathers

@SciTools/peloton We are planning to talk about this on the 3rd July.

@pp-mo and @trexfeathers met on 3rd July. This review is the summary of that meeting.

@larsbarring we really appreciate you taking the time to make this suggestion. It's obviously bad that it's taken us this long to give it any attention. If your priorities have shifted and you are unable to action our comments then please let us know and we'll pick it up.

To prompt more attention from the team, we're adding this to the 3.11 milestone.

trexfeathers · 2024-07-17T11:03:33Z

tools/generate_std_names.py

+        '_ACCEPT = "accept"\n'
+        '_WARN = "warn"\n'
+        '_REPLACE ="replace"\n'
+        '_ALTERNATIVE_MODES = [_ACCEPT, _WARN, _REPLACE]\n'
+        '_DEFAULT = "warn"\n'
+        '_MODE = _DEFAULT\n\n'


We can see what you are trying to achieve here, but we would like to achieve it using Iris' existing structure instead:

FUTURE flag

Replacement is introduced as an opt-in setting. In future - typically the next major release - it will switch to being opt-out.

Always warn if replacing

Since you made the proposal, we have agreed a global approach to Iris warnings: Filtering Warnings — Iris 3.9.0 documentation (scitools-iris.readthedocs.io). The idea is that Iris should warn for EVERYTHING that might be a problem for a user, and the user applies warning filters if they don't wish to see a specific type of warning.

trexfeathers · 2024-07-17T11:14:32Z

lib/iris/common/mixin.py

We should consider whether the rename() method needs a switch to accept aliases, or some other equivalent granular user control.

@pp-mo I can't recall why using a FUTURE flag would not be enough user control on its own?

trexfeathers · 2024-07-17T11:17:55Z

tools/generate_std_names.py

+            version["table_name"] = "USER-StdNameTable"
+    prettydict(outfile, "VERSION", version)
+    version_string = "-".join(version[k] for k in ["table_name", "version_number"])
+    outfile.write(f'CONVENTIONS_STRING = "{version_string}"\n\n')


Note that this was sorted in #5423, albeit in a different way.

Presumably no longer necessary in this PR?

trexfeathers · 2024-07-17T11:24:00Z

lib/iris/common/mixin.py

For consistency with existing Iris code, we believe a new attribute should be introduced: alias_standard_name, used in a similar way to invalid_standard_name.

This gives users the level of information and control that they are used to.

invalid_standard_name example

iris/lib/iris/fileformats/_nc_load_rules/helpers.py

Lines 463 to 470 in 8604bb8

if standard_name is not None:

try:

cube.standard_name = _get_valid_standard_name(standard_name)

except ValueError:

if cube.long_name is not None:

cube.attributes["invalid_standard_name"] = standard_name

else:

cube.long_name = standard_name

larsbarring mentioned this pull request May 11, 2023

Handling of Standard name aliases #5257

Open

larsbarring added 8 commits May 15, 2023 14:31

fix to xml file (remove ctrl-M), cf CF-conventions/discuss/SciTools#229:

d8a4325

https://github.com/cf-convention/discuss/issues/229 Standard names: hard to spot "typo" in `surface_upward_mass_flux_of_methane_due_to_emission_from_fires`

added standard name handling functions in standard_name_table.py

e94d654

Updated mixin.py to use new standard name handling

dc98905

Naively placed 'import iris.std_name_table' in iris.__init__

7b5467c

Cleanup of structure

1ab8fd9

aligned common/mixin.py with original

235ac02

Added license header to std_name_table.py

d2216e7

larsbarring force-pushed the handle_std_names branch from 57fed3f to d2216e7 Compare May 15, 2023 12:37

trexfeathers requested a review from pp-mo November 8, 2023 10:37

schlunma mentioned this pull request May 2, 2024

Multi-model statistics on changed standard names through CMIP projects ESMValGroup/ESMValCore#1985

Open

pp-mo self-assigned this Jul 5, 2024

trexfeathers requested changes Jul 17, 2024

View reviewed changes

LisaBock mentioned this pull request Sep 23, 2024

Adding figures of Bock and Lauer (2024) ESMValGroup/ESMValTool#3526

Open

11 tasks

HGWright added this to the v3.12 milestone Oct 9, 2024

HGWright unassigned pp-mo Oct 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

POC: Handle std names and aliases (#5257) #5313

POC: Handle std names and aliases (#5257) #5313

larsbarring commented May 11, 2023

codecov bot commented May 11, 2023 •

edited

Loading

CLAassistant commented Apr 8, 2024 •

edited

Loading

pp-mo commented May 1, 2024

larsbarring commented May 1, 2024

pp-mo commented May 1, 2024

larsbarring commented May 1, 2024

schlunma commented May 2, 2024 •

edited

Loading

bouweandela commented Jun 6, 2024

HGWright commented Jun 12, 2024

trexfeathers left a comment

trexfeathers Jul 17, 2024

trexfeathers Jul 17, 2024

trexfeathers Jul 17, 2024

trexfeathers Jul 17, 2024

	if standard_name is not None:
	try:
	cube.standard_name = _get_valid_standard_name(standard_name)
	except ValueError:
	if cube.long_name is not None:
	cube.attributes["invalid_standard_name"] = standard_name
	else:
	cube.long_name = standard_name

POC: Handle std names and aliases (#5257) #5313

Are you sure you want to change the base?

POC: Handle std names and aliases (#5257) #5313

Conversation

larsbarring commented May 11, 2023

🚀 Pull Request

Description

codecov bot commented May 11, 2023 • edited Loading

Codecov Report

CLAassistant commented Apr 8, 2024 • edited Loading

pp-mo commented May 1, 2024

larsbarring commented May 1, 2024

pp-mo commented May 1, 2024

larsbarring commented May 1, 2024

schlunma commented May 2, 2024 • edited Loading

bouweandela commented Jun 6, 2024

HGWright commented Jun 12, 2024

trexfeathers left a comment

Choose a reason for hiding this comment

trexfeathers Jul 17, 2024

Choose a reason for hiding this comment

FUTURE flag

Always warn if replacing

trexfeathers Jul 17, 2024

Choose a reason for hiding this comment

trexfeathers Jul 17, 2024

Choose a reason for hiding this comment

trexfeathers Jul 17, 2024

Choose a reason for hiding this comment

invalid_standard_name example

codecov bot commented May 11, 2023 •

edited

Loading

CLAassistant commented Apr 8, 2024 •

edited

Loading

schlunma commented May 2, 2024 •

edited

Loading

`FUTURE` flag

`invalid_standard_name` example