MOBT-661: Required changes for precipitation phase accumulations #1997

brhooper · 2024-05-15T07:08:52Z

Related to: https://github.com/metoppv/mo-blue-team/issues/661, https://github.com/metoppv/mo-blue-team/issues/671
Acceptance test data: metoppv/improver_test_data#48

Description
This PR includes a few changes which are required for the work to produce precipitation phase accumulations over longer periods.

Included changes are as follows:

Add option for 'realization' tie-breaking within ECC. This resolves ties within the dependence template by assigning values to the highest realization number.
Factor out code which enables recycling realizations in the ECC dependence template.
Create a CLI for the above recycling code to enable extending the realization dimension on phase probability cubes. This ensures realization recycling is done using a consistent method, which is important for precipitation phase accumulation diagnostics for reasons laid out in this comment (https://github.com/MetOffice/improver_aux/pull/420#issuecomment-2051865950).
Make one of the checks in the combine plugin easier to pass - test will no longer fail if coordinates on input cubes differ in the var_name, except if the difference is on a threshold coordinate. This mirrors, and was necessitated by, the functionality of the MergeCubes plugin, which removes this attribute from coordinates.
Add functionality to 'recursive_filter' plugin to allow application to cubes containing different masks on each spatial slice. Also adds this functionality to the corresponding CLI.
Add unit and acceptance tests where required for each of the above.

Testing:

Ran tests and they passed OK
Added new tests for the new feature(s)

codecov · 2024-05-15T07:29:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.39%. Comparing base (84a8944) to head (523ebf6).
Report is 35 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1997      +/-   ##
==========================================
- Coverage   98.39%   98.39%   -0.01%     
==========================================
  Files         124      133       +9     
  Lines       12212    13060     +848     
==========================================
+ Hits        12016    12850     +834     
- Misses        196      210      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-09-21T01:34:58Z

In order to maintain a backlog of relevant PRs, we automatically label them as stale after 60 days of inactivity.

If this PR is still important to you, then please comment on this PR and the stale label will be removed.

Otherwise this PR will be automatically closed in 30 days time.

brhooper · 2024-09-26T07:07:41Z

Commenting to remove 'staleness'.

gavinevans

Thanks @brhooper 👍

I've added some comments and suggestions.

improver/nbhood/recursive_filter.py

improver/utilities/cube_manipulation.py

improver_tests/acceptance/test_generate_realizations.py

improver_tests/utilities/cube_manipulation/test_manipulate_realization_dimension.py

improver/cli/manipulate_realization_dimension.py

improver/ensemble_copula_coupling/ensemble_copula_coupling.py

improver_tests/acceptance/SHA256SUMS

gavinevans

Thanks @brhooper 👍

I've happy to approve these changes, subject to the GitHub Actions failures being addressed.

…ities. Add unit tests.

…ts of new functionality.

…Add acceptance test and data for this functionality. Update checksums. Change numpy broadcasting method in plugin.

…ding acceptance tests. Update checksums.

…dimension coordinate var_name attributes does not cause an error, except where the var_name is different on a threshold coordinate. Change made to better work with the functionality of MergeCubes, which removes var_name from non-threshold dimension coordinates.

… the input cube. Update checksums for new data for this test.

…ations.

brhooper · 2024-10-15T07:19:33Z

Thanks @brhooper 👍

I've happy to approve these changes, subject to the GitHub Actions failures being addressed.

I've pulled the changes from this PR into my PR. The GIthub actions now succeed as expected following this fix.

bayliffe

Incredibly minor questions / comments.

bayliffe · 2024-11-04T08:34:37Z

improver/cli/manipulate_n_realizations.py

+
+    output = manipulate_n_realizations(cube, n_realizations)
+
+    return output


I haven't really got a map in my head of the way you'll implement this in the suite, but I'm a little surprised this method got a CLI. Are you intending to call it as a preliminary step in the suite chain, rather than it being something that is used internally within ECC?

Okay, you do call this without the CLI. Is this just for completeness, an envisaged future use, or just to allow for acceptance tests?

Good question. I think I added the CLI out of habit to be honest.

I can't think of a use case for the manipulate-n-realizations CLI at the moment and the acceptance tests don't test anything which isn't covered by the unit tests. As such, I've removed the CLI and associated acceptance tests.

bayliffe · 2024-11-04T09:00:53Z

improver/cube_combiner.py

@@ -222,8 +223,10 @@ def _check_dimensions_match(
        Raises:
            ValueError: If dimension coordinates do not match
        """
-        ref_coords = cube_list[0].coords(dim_coords=True)
-        for cube in cube_list[1:]:
+        test_cube_list = iris.cube.CubeList(cube_list.copy())


The doc-string probably now needs to specify that this check excludes the var_names of the coordinates.

Doc-string updated

bayliffe · 2024-11-04T09:31:02Z

improver_tests/ensemble_copula_coupling/test_EnsembleReordering.py

+        calibrated_cube = self.cube_2d.copy(data=calibrated_data)
+
+        result = Plugin().rank_ecc(calibrated_cube, raw_cube, random_seed=1)
+        self.assertArrayAlmostEqual(result.data, result_data)


Any particular reason this test was modified? Is it simply to differentiate it to the test below?

Yes, I modified the test so that result_data was different for the 2 tests. My aim was to demonstrate the difference between using random and realisation tie-breaking.

…ceptance tests. Add additional information to _check_dimensions_match doc-string.

… data.

bayliffe

Thanks for all of this Ben.

brhooper mentioned this pull request May 15, 2024

MOBT-661: Required changes for precipitation phase accumulations metoppv/improver_test_data#48

Merged

brhooper changed the title ~~Required changes for precipitation phase accumulations~~ MOBT-661: Required changes for precipitation phase accumulations May 15, 2024

bayliffe self-assigned this May 16, 2024

brhooper assigned brhooper and unassigned bayliffe May 31, 2024

brhooper added the Paused label Jul 22, 2024

github-actions bot added the Stale label Sep 21, 2024

brhooper removed Stale Paused labels Sep 26, 2024

brhooper force-pushed the mobt_671_ecc_extensions branch 2 times, most recently from 873974b to 46ccc84 Compare October 3, 2024 13:38

brhooper removed their assignment Oct 3, 2024

gavinevans self-assigned this Oct 3, 2024

gavinevans requested changes Oct 4, 2024

View reviewed changes

gavinevans assigned brhooper and unassigned gavinevans Oct 4, 2024

brhooper assigned gavinevans and unassigned brhooper Oct 10, 2024

gavinevans previously approved these changes Oct 10, 2024

View reviewed changes

gavinevans assigned brhooper and unassigned gavinevans and brhooper Oct 10, 2024

brhooper added 5 commits October 15, 2024 07:46

Refactor realization dimension extension into own function under util…

ca1154e

…ities. Add unit tests.

Add new tie-breaking method to ECC implementation. Add associated tes…

2f5293e

…ts of new functionality.

Update generate_realizations CLI to enable new plugin functionality. …

0f91ad1

…Add acceptance test and data for this functionality. Update checksums. Change numpy broadcasting method in plugin.

Create CLI for manipulate_realization_dimension plugin. Add correspon…

0d34967

…ding acceptance tests. Update checksums.

brhooper added 9 commits October 15, 2024 07:46

Formatting changes. Remove unnecessary module import.

e1ea2a0

Update license statement in new files.

a590c91

Change copyright statement in files to new, shorter format.

1606d13

Remove unnecessary comment.

1301912

Add new accpetance test for recursive filter using a variable mask on…

ed08936

… the input cube. Update checksums for new data for this test.

Formatting changes.

1785fef

Changes following review.

bf8df22

Formatting changes.

51a17d6

Rename manipulate_realization_dimension plugin to manipulate_n_realiz…

f7aff40

…ations.

brhooper force-pushed the mobt_671_ecc_extensions branch from b37cf83 to f7aff40 Compare October 15, 2024 07:09

brhooper assigned bayliffe Oct 28, 2024

bayliffe reviewed Nov 4, 2024

View reviewed changes

bayliffe assigned brhooper and unassigned bayliffe Nov 4, 2024

brhooper added 2 commits November 7, 2024 07:38

Changes following review. Remove manipulate_n_realizations CLI and ac…

f5ed88b

…ceptance tests. Add additional information to _check_dimensions_match doc-string.

Update acceptance test data checksums.

85787a2

brhooper dismissed gavinevans’s stale review via 85787a2 November 7, 2024 11:51

brhooper assigned bayliffe and unassigned brhooper Nov 7, 2024

Update acceptance test data checksums following another change to the…

523ebf6

… data.

bayliffe approved these changes Nov 8, 2024

View reviewed changes

bayliffe merged commit 3c21fff into metoppv:master Nov 8, 2024
10 checks passed

bayliffe assigned brhooper and unassigned bayliffe Nov 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MOBT-661: Required changes for precipitation phase accumulations #1997

MOBT-661: Required changes for precipitation phase accumulations #1997

brhooper commented May 15, 2024 •

edited

Loading

codecov bot commented May 15, 2024 •

edited

Loading

github-actions bot commented Sep 21, 2024

brhooper commented Sep 26, 2024

gavinevans left a comment

gavinevans left a comment

brhooper commented Oct 15, 2024

bayliffe left a comment

bayliffe Nov 4, 2024

bayliffe Nov 4, 2024

brhooper Nov 7, 2024

bayliffe Nov 4, 2024

brhooper Nov 7, 2024

bayliffe Nov 4, 2024

brhooper Nov 7, 2024

bayliffe left a comment


		output = manipulate_n_realizations(cube, n_realizations)

		return output

MOBT-661: Required changes for precipitation phase accumulations #1997

MOBT-661: Required changes for precipitation phase accumulations #1997

Conversation

brhooper commented May 15, 2024 • edited Loading

codecov bot commented May 15, 2024 • edited Loading

Codecov Report

github-actions bot commented Sep 21, 2024

brhooper commented Sep 26, 2024

gavinevans left a comment

Choose a reason for hiding this comment

gavinevans left a comment

Choose a reason for hiding this comment

brhooper commented Oct 15, 2024

bayliffe left a comment

Choose a reason for hiding this comment

bayliffe Nov 4, 2024

Choose a reason for hiding this comment

bayliffe Nov 4, 2024

Choose a reason for hiding this comment

brhooper Nov 7, 2024

Choose a reason for hiding this comment

bayliffe Nov 4, 2024

Choose a reason for hiding this comment

brhooper Nov 7, 2024

Choose a reason for hiding this comment

bayliffe Nov 4, 2024

Choose a reason for hiding this comment

brhooper Nov 7, 2024

Choose a reason for hiding this comment

bayliffe left a comment

Choose a reason for hiding this comment

brhooper commented May 15, 2024 •

edited

Loading

codecov bot commented May 15, 2024 •

edited

Loading