Add support for Iris 3.0 #1507
Merged
Add support for Iris 3.0 #1507
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #1507 +/- ##
==========================================
+ Coverage 97.84% 97.93% +0.08%
==========================================
Files 106 106
Lines 9472 9447 -25
==========================================
- Hits 9268 9252 -16
+ Misses 204 195 -9
Continue to review full report at Codecov.
|
|
Although the unit tests are passing, 36 acceptance tests are failing with errors like: _______________________ test_normal_point_by_point_sites _______________________
[gw4] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0')
def test_normal_point_by_point_sites(tmp_path):
"""Test using a normal distribution when coefficients have been calculated
independently at each site (initial guess and minimisation)."""
kgo_dir = acc.kgo_root() / "apply-emos-coefficients/sites/point_by_point"
kgo_path = kgo_dir / "kgo.nc"
input_path = kgo_dir / ".." / "input.nc"
emos_est_path = kgo_dir / "coefficients.nc"
output_path = tmp_path / "output.nc"
args = [
input_path,
emos_est_path,
"--random-seed",
"0",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path, atol=LOOSE_TOLERANCE)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_apply_emos_coefficients.py:99: AssertionError
----------------------------- Captured stdout call -----------------------------
improver apply-emos-coefficients /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/../input.nc /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/coefficients.nc --random-seed 0 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0/output.ncIt seems some datasets are missing attributes, mostly units, e.g. Here is the full output from ============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/dmentipl/repos/improver, configfile: pytest.ini, testpaths: improver_tests
plugins: cov-2.12.0, forked-1.3.0, xdist-2.2.1
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I
gw0 [347] / gw1 [347] / gw2 [347] / gw3 [347] / gw4 [347] / gw5 [347] / gw6 [347] / gw7 [347] / gw8 [347] / gw9 [347] / gw10 [347] / gw11 [347] / gw12 [347] / gw13 [347] / gw14 [347] / gw15 [347]
..............F................F.....s.....s............................ [ 20%]
.........................F......F..F.....F.F....F..F..............F..... [ 41%]
.........F...FF........F.FF..F..F..FF....FFF..FF..FF..FssF........F..... [ 62%]
..........................x.......x..................................... [ 82%]
..........................F...........FF.....F...FF........ [100%]
=================================== FAILURES ===================================
__________________________________ test_basic __________________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_basic0')
def test_basic(tmp_path):
"""Test basic create grid with halo"""
kgo_dir = acc.kgo_root() / "create-grid-with-halo/basic"
kgo_path = kgo_dir / "kgo.nc"
input_path = kgo_dir / "source_grid.nc"
output_path = tmp_path / "output.nc"
args = [input_path, "--output", output_path]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /grid_with_halo - ['units']
improver_tests/acceptance/test_create_grid_with_halo.py:50: AssertionError
----------------------------- Captured stdout call -----------------------------
improver create-grid-with-halo /mnt/data/dmentipl/improver_tests/create-grid-with-halo/basic/source_grid.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_basic0/output.nc
_______________________ test_normal_point_by_point_sites _______________________
[gw4] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0')
def test_normal_point_by_point_sites(tmp_path):
"""Test using a normal distribution when coefficients have been calculated
independently at each site (initial guess and minimisation)."""
kgo_dir = acc.kgo_root() / "apply-emos-coefficients/sites/point_by_point"
kgo_path = kgo_dir / "kgo.nc"
input_path = kgo_dir / ".." / "input.nc"
emos_est_path = kgo_dir / "coefficients.nc"
output_path = tmp_path / "output.nc"
args = [
input_path,
emos_est_path,
"--random-seed",
"0",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path, atol=LOOSE_TOLERANCE)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_apply_emos_coefficients.py:99: AssertionError
----------------------------- Captured stdout call -----------------------------
improver apply-emos-coefficients /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/../input.nc /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/coefficients.nc --random-seed 0 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0/output.nc
____________________________ test_nearest[uk-ukvx] _____________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_uk_ukvx_0')
domain = 'uk', model = 'ukvx'
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest(tmp_path, domain, model):
"""Test basic neighbour finding"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [orography_path, landmask_path, sites_path, "--output", output_path]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:59: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_uk_ukvx_0/output.nc
_________________________ test_nearest[global-global] __________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_global_global_0')
domain = 'global', model = 'global'
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest(tmp_path, domain, model):
"""Test basic neighbour finding"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [orography_path, landmask_path, sites_path, "--output", output_path]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:59: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_global_global_0/output.nc
________________________________ test_halo_size ________________________________
[gw2] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw2/test_halo_size0')
def test_halo_size(tmp_path):
"""Test basic create grid with halo"""
kgo_dir = acc.kgo_root() / "create-grid-with-halo/halo_size"
kgo_path = kgo_dir / "kgo.nc"
input_path = kgo_dir / "../basic/source_grid.nc"
output_path = tmp_path / "output.nc"
args = [input_path, "--halo-radius", "75000", "--output", output_path]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /grid_with_halo - ['units']
improver_tests/acceptance/test_create_grid_with_halo.py:61: AssertionError
----------------------------- Captured stdout call -----------------------------
improver create-grid-with-halo /mnt/data/dmentipl/improver_tests/create-grid-with-halo/halo_size/../basic/source_grid.nc --halo-radius 75000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw2/test_halo_size0/output.nc
______________________________ test_invalid_site _______________________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_invalid_site0')
def test_invalid_site(tmp_path):
"""Test invalid global site coordinates"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / "outputs/nearest_global_invalid_site_kgo.nc"
sites_path = kgo_dir / "inputs/global_sites_invalid.json"
orography_path = kgo_dir / "inputs/global_orography.nc"
landmask_path = kgo_dir / "inputs/global_landmask.nc"
output_path = tmp_path / "output.nc"
args = [orography_path, landmask_path, sites_path, "--output", output_path]
with pytest.warns(UserWarning, match=".*outside the grid.*"):
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:223: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites_invalid.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_invalid_site0/output.nc
___________________________ test_coord_beyond_bounds ___________________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_coord_beyond_bounds0')
def test_coord_beyond_bounds(tmp_path):
"""Test coordinates up to 360 degrees"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / "outputs/nearest_global_kgo.nc"
sites_path = kgo_dir / "inputs/global_sites_360.json"
orography_path = kgo_dir / "inputs/global_orography.nc"
landmask_path = kgo_dir / "inputs/global_landmask.nc"
output_path = tmp_path / "output.nc"
args = [orography_path, landmask_path, sites_path, "--output", output_path]
run_cli(args)
> acc.compare(output_path, kgo_path, exclude_vars=["longitude"])
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:236: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites_360.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_coord_beyond_bounds0/output.nc
______________________ test_unset_wmo_ids_with_unique_ids ______________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_unset_wmo_ids_with_unique0')
def test_unset_wmo_ids_with_unique_ids(tmp_path):
"""Test sites with missing WMO ID numbers, but with valid unique IDs for
all sites"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / "outputs/nearest_uk_kgo_some_unset_wmo_ids_unique_ids.nc"
sites_path = kgo_dir / "inputs/uk_sites_missing_wmo_ids_with_unique_ids.json"
orography_path = kgo_dir / "inputs/ukvx_orography.nc"
landmask_path = kgo_dir / "inputs/ukvx_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--unique-site-id-key",
"met_office_site_id",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:258: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites_missing_wmo_ids_with_unique_ids.json --unique-site-id-key met_office_site_id --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_unset_wmo_ids_with_unique0/output.nc
__________________________ test_nearest_land[uk-ukvx] __________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_uk_ukvx_0')
domain = 'uk', model = 'ukvx'
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest_land(tmp_path, domain, model):
"""Test neighbour finding with a land constraint"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_land_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--land-constraint",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:80: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --land-constraint --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_uk_ukvx_0/output.nc
_______________________ test_nearest_land[global-global] _______________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_global_globa0')
domain = 'global', model = 'global'
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest_land(tmp_path, domain, model):
"""Test neighbour finding with a land constraint"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_land_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--land-constraint",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:80: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --land-constraint --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_global_globa0/output.nc
____________ test_normal_point_by_point_default_initial_guess_sites ____________
[gw13] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw13/test_normal_point_by_point_def0')
@pytest.mark.slow
def test_normal_point_by_point_default_initial_guess_sites(tmp_path):
"""
Test estimate-emos-coefficients for diagnostic with assumed
normal distribution where coefficients are computed independently at each
site location (minimisation only).
"""
kgo_dir = acc.kgo_root() / "estimate-emos-coefficients/normal/sites"
kgo_path = kgo_dir / "point_by_point_default_initial_guess" / "kgo.nc"
history_path = kgo_dir / "history/*.nc"
truth_path = kgo_dir / "truth/*.nc"
output_path = tmp_path / "output.nc"
est_emos_tol = str(0.01)
compare_emos_tolerance = 0.1
args = [
history_path,
truth_path,
"--distribution",
"norm",
"--truth-attribute",
"mosg__model_configuration=uk_det",
"--tolerance",
est_emos_tol,
"--point-by-point",
"--use-default-initial-guess",
"--output",
output_path,
]
run_cli(args)
acc.compare(
> output_path, kgo_path, atol=compare_emos_tolerance, rtol=compare_emos_tolerance
)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_estimate_emos_coefficients.py:366: AssertionError
----------------------------- Captured stdout call -----------------------------
improver estimate-emos-coefficients '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/history/*.nc' '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/truth/*.nc' --distribution norm --truth-attribute mosg__model_configuration=uk_det --tolerance 0.01 --point-by-point --use-default-initial-guess --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw13/test_normal_point_by_point_def0/output.nc
_______________________ test_nearest_minimum_dz[uk-ukvx] _______________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_uk_ukv0')
domain = 'uk', model = 'ukvx'
@pytest.mark.slow
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest_minimum_dz(tmp_path, domain, model):
"""Test neighbour finding with minimum altitude difference"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_minimum_dz_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--similar-altitude",
"--search-radius",
"50000",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:104: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_uk_ukv0/output.nc
__________________________ test_all_methods[uk-ukvx] ___________________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_uk_ukvx_0')
domain = 'uk', model = 'ukvx'
@pytest.mark.slow
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_all_methods(tmp_path, domain, model):
"""Test neighbour finding using all methods"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/all_methods_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--all-methods",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /neighbour_selection_method_name - ['units']
improver_tests/acceptance/test_neighbour_finding.py:153: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --all-methods --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_uk_ukvx_0/output.nc
__ test_nearest_uk[all_methods_uk.nc-extra_args0-nearest_uk_temperatures.nc] ___
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk0')
neighbour_cube = 'all_methods_uk.nc', extra_args = []
kgo_file = 'nearest_uk_temperatures.nc'
@pytest.mark.parametrize(
"neighbour_cube,extra_args,kgo_file",
(
("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
),
)
def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
"""Test spot extraction using nearest location"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
kgo_path = kgo_dir / f"outputs/{kgo_file}"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--output",
output_path,
*extra_args,
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk0/output.nc --new-title 'IMPROVER UK Spot Values'
___ test_nearest_uk[all_methods_uk.nc-extra_args1-mindz_uk_temperatures.nc] ____
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk1')
neighbour_cube = 'all_methods_uk.nc', extra_args = ['--similar-altitude']
kgo_file = 'mindz_uk_temperatures.nc'
@pytest.mark.parametrize(
"neighbour_cube,extra_args,kgo_file",
(
("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
),
)
def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
"""Test spot extraction using nearest location"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
kgo_path = kgo_dir / f"outputs/{kgo_file}"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--output",
output_path,
*extra_args,
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk1/output.nc --similar-altitude --new-title 'IMPROVER UK Spot Values'
_________________________ test_lapse_rate_adjusted_uk __________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_lapse_rate_adjusted_uk0')
def test_lapse_rate_adjusted_uk(tmp_path):
"""Test spot extraction with lapse rate adjustment"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
lapse_path = kgo_dir / "inputs/ukvx_lapse_rate.nc"
kgo_path = kgo_dir / "outputs/lapse_rate_adjusted_uk_temperatures.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
lapse_path,
"--output",
output_path,
"--apply-lapse-rate-correction",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:94: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_lapse_rate.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_lapse_rate_adjusted_uk0/output.nc --apply-lapse-rate-correction --new-title 'IMPROVER UK Spot Values'
_ test_nearest_uk[all_methods_uk_unique_ids.nc-extra_args2-nearest_uk_temperatures_unique_ids.nc] _
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk2')
neighbour_cube = 'all_methods_uk_unique_ids.nc', extra_args = []
kgo_file = 'nearest_uk_temperatures_unique_ids.nc'
@pytest.mark.parametrize(
"neighbour_cube,extra_args,kgo_file",
(
("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
),
)
def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
"""Test spot extraction using nearest location"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
kgo_path = kgo_dir / f"outputs/{kgo_file}"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--output",
output_path,
*extra_args,
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk_unique_ids.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk2/output.nc --new-title 'IMPROVER UK Spot Values'
_____________________________ test_new_spot_title ______________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_new_spot_title0')
def test_new_spot_title(tmp_path):
"""Test spot extraction with external JSON metadata"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
kgo_path = kgo_dir / "outputs/nearest_uk_temperatures_amended_metadata.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--new-title",
UK_SPOT_TITLE,
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:179: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --new-title 'IMPROVER UK Spot Values' --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_new_spot_title0/output.nc
___________________________ test_lapse_rate_mismatch ___________________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_lapse_rate_mismatch0')
def test_lapse_rate_mismatch(tmp_path):
"""Test lapse rate adjustment mismatch between datasets"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
lapse_path = kgo_dir / "inputs/ukvx_lapse_rate_2m.nc"
kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
lapse_path,
"--output",
output_path,
"--apply-lapse-rate-correction",
"--new-title",
UK_SPOT_TITLE,
]
with pytest.warns(UserWarning, match=".*height.*not adjusted.*"):
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:139: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_lapse_rate_2m.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_lapse_rate_mismatch0/output.nc --apply-lapse-rate-correction --new-title 'IMPROVER UK Spot Values'
_______________________ test_percentile_percentile_input _______________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_percentile_inp0')
def test_percentile_percentile_input(tmp_path):
"""Test extracting percentiles from percentile input"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
threshold_path = kgo_dir / "inputs/enukx_temperature_percentiles.nc"
kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
threshold_path,
"--output",
output_path,
"--extract-percentiles",
"50",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:261: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_percentiles.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_percentile_inp0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
________________________ test_percentile_deterministic _________________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic0')
def test_percentile_deterministic(tmp_path):
"""Test extracting percentiles from deterministic input"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--output",
output_path,
"--extract-percentiles",
"50",
"--new-title",
UK_SPOT_TITLE,
]
with pytest.warns(
UserWarning, match="Diagnostic cube is not a known probabilistic type."
):
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:303: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
__________________________________ test_spot ___________________________________
[gw5] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw5/test_spot0')
def test_spot(tmp_path):
"""Test extraction from spot file"""
kgo_dir = acc.kgo_root() / "extract/sites"
kgo_path = kgo_dir / "kgo_spot.nc"
input_path = kgo_dir / "input_spot.nc"
output_path = tmp_path / "output.nc"
args = [
input_path,
"--constraints",
"wmo_id=['700', '845', '996', '3346', '3382']",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_extract.py:168: AssertionError
----------------------------- Captured stdout call -----------------------------
improver extract /mnt/data/dmentipl/improver_tests/extract/sites/input_spot.nc --constraints 'wmo_id=['"'"'700'"'"', '"'"'845'"'"', '"'"'996'"'"', '"'"'3346'"'"', '"'"'3382'"'"']' --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw5/test_spot0/output.nc
__________________ test_multiple_percentile_thresholded_input __________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_thres0')
def test_multiple_percentile_thresholded_input(tmp_path):
"""Test extracting multiple percentiles from thresholded input"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
threshold_path = kgo_dir / "inputs/enukx_temperature_thresholds.nc"
kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
threshold_path,
"--output",
output_path,
"--extract-percentiles",
"25, 50, 75",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:354: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_thres0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
_____________________ test_percentile_deterministic_quiet ______________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic_0')
def test_percentile_deterministic_quiet(tmp_path):
"""Test extracting percentiles from deterministic input. In this case the
--suppress-warnings flag is enabled. This excludes the warning raised when
spot-extract is set to extract percentiles and used with deterministic data.
This is intended to reduce output in logs when this warning is expected and
thus not useful."""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--output",
output_path,
"--extract-percentiles",
"50",
"--new-title",
UK_SPOT_TITLE,
"--suppress-warnings",
]
with pytest.warns(None) as collected_warns:
run_cli(args)
msg = "Diagnostic cube is not a known probabilistic type."
assert all([msg not in str(warning.message) for warning in collected_warns])
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:333: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic_0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values' --suppress-warnings
__________________ test_multiple_percentile_percentile_input ___________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_perce0')
def test_multiple_percentile_percentile_input(tmp_path):
"""Test extracting multiple percentiles from percentile input"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
threshold_path = kgo_dir / "inputs/enukx_temperature_percentiles.nc"
kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
threshold_path,
"--output",
output_path,
"--extract-percentiles",
"25, 50, 75",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:375: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_percentiles.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_perce0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
__________________________ test_multiple_constraints ___________________________
[gw9] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_multiple_constraints0')
def test_multiple_constraints(tmp_path):
"""Test use of multiple constraints"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
kgo_path = kgo_dir / "outputs/mindz_land_constraint_uk_temperatures.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--output",
output_path,
"--similar-altitude",
"--land-constraint",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:219: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_multiple_constraints0/output.nc --similar-altitude --land-constraint --new-title 'IMPROVER UK Spot Values'
______________________ test_percentile_realization_input _______________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_realization_in0')
def test_percentile_realization_input(tmp_path):
"""Test extracting percentiles from realization input"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
realization_path = kgo_dir / "inputs/enukx_temperature_realizations.nc"
kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
realization_path,
"--output",
output_path,
"--extract-percentiles",
"50",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:396: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_realizations.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_realization_in0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
______________________ test_percentile_thresholded_input _______________________
[gw9] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_percentile_thresholded_in0')
def test_percentile_thresholded_input(tmp_path):
"""Test extracting percentiles from thresholded input"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
threshold_path = kgo_dir / "inputs/enukx_temperature_thresholds.nc"
kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
threshold_path,
"--output",
output_path,
"--extract-percentiles",
"50",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:240: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_percentile_thresholded_in0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
__________________ test_multiple_percentile_realization_input __________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_multiple_percentile_reali0')
def test_multiple_percentile_realization_input(tmp_path):
"""Test extracting multiple percentiles from realization input"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
realization_path = kgo_dir / "inputs/enukx_temperature_realizations.nc"
kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
realization_path,
"--output",
output_path,
"--extract-percentiles",
"25, 50, 75",
"--new-title",
UK_SPOT_TITLE,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:417: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_realizations.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_multiple_percentile_reali0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
_______________ test_percentile_from_threshold_with_realizations _______________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_from_threshold0')
def test_percentile_from_threshold_with_realizations(tmp_path):
"""Test requesting a percentile from a cube with thresholds where the realizations
need collapsing first"""
kgo_dir = acc.kgo_root() / "spot-extract"
neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
diag_path = kgo_dir / "inputs/enukx_preciprate_realizations_thresholds.nc"
kgo_path = kgo_dir / "outputs/with_realization_collapse.nc"
output_path = tmp_path / "output.nc"
args = [
neighbour_path,
diag_path,
"--output",
output_path,
"--realization-collapse",
"--extract-percentiles",
"50",
"--new-title",
"MOGREPS-UK Spot Values",
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_spot_extract.py:480: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_preciprate_realizations_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_from_threshold0/output.nc --realization-collapse --extract-percentiles 50 --new-title 'MOGREPS-UK Spot Values'
_______________________ test_normal_point_by_point_sites _______________________
[gw10] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw10/test_normal_point_by_point_sit0')
@pytest.mark.slow
def test_normal_point_by_point_sites(tmp_path):
"""
Test estimate-emos-coefficients for diagnostic with assumed
normal distribution where coefficients are computed independently at each
site location (initial guess and minimisation).
"""
kgo_dir = acc.kgo_root() / "estimate-emos-coefficients/normal/sites"
kgo_path = kgo_dir / "point_by_point" / "kgo.nc"
history_path = kgo_dir / "history/*.nc"
truth_path = kgo_dir / "truth/*.nc"
output_path = tmp_path / "output.nc"
est_emos_tol = str(0.01)
compare_emos_tolerance = 0.1
args = [
history_path,
truth_path,
"--distribution",
"norm",
"--truth-attribute",
"mosg__model_configuration=uk_det",
"--tolerance",
est_emos_tol,
"--point-by-point",
"--output",
output_path,
]
run_cli(args)
acc.compare(
> output_path, kgo_path, atol=compare_emos_tolerance, rtol=compare_emos_tolerance
)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_estimate_emos_coefficients.py:296: AssertionError
----------------------------- Captured stdout call -----------------------------
improver estimate-emos-coefficients '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/history/*.nc' '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/truth/*.nc' --distribution norm --truth-attribute mosg__model_configuration=uk_det --tolerance 0.01 --point-by-point --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw10/test_normal_point_by_point_sit0/output.nc
____________________ test_nearest_minimum_dz[global-global] ____________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_global0')
domain = 'global', model = 'global'
@pytest.mark.slow
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest_minimum_dz(tmp_path, domain, model):
"""Test neighbour finding with minimum altitude difference"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_minimum_dz_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--similar-altitude",
"--search-radius",
"50000",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:104: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_global0/output.nc
____________________ test_nearest_land_minimum_dz[uk-ukvx] _____________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_u0')
domain = 'uk', model = 'ukvx'
@pytest.mark.slow
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest_land_minimum_dz(tmp_path, domain, model):
"""Test neighbour finding with land constraint plus min altitude diff"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_land_constraint_minimum_dz_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--land-constraint",
"--similar-altitude",
"--search-radius",
"50000",
"--output",
output_path,
]
if domain == "uk":
args += ["--node-limit", "100"]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:131: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --land-constraint --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_u0/output.nc --node-limit 100
_________________ test_nearest_land_minimum_dz[global-global] __________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_g0')
domain = 'global', model = 'global'
@pytest.mark.slow
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_nearest_land_minimum_dz(tmp_path, domain, model):
"""Test neighbour finding with land constraint plus min altitude diff"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/nearest_land_constraint_minimum_dz_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--land-constraint",
"--similar-altitude",
"--search-radius",
"50000",
"--output",
output_path,
]
if domain == "uk":
args += ["--node-limit", "100"]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /wmo_id - ['units']
improver_tests/acceptance/test_neighbour_finding.py:131: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --land-constraint --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_g0/output.nc
_______________________ test_all_methods[global-global] ________________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_global_global0')
domain = 'global', model = 'global'
@pytest.mark.slow
@pytest.mark.parametrize("domain,model", UK_GLOBAL)
def test_all_methods(tmp_path, domain, model):
"""Test neighbour finding using all methods"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
kgo_path = kgo_dir / f"outputs/all_methods_{domain}_kgo.nc"
sites_path = kgo_dir / f"inputs/{domain}_sites.json"
orography_path = kgo_dir / f"inputs/{model}_orography.nc"
landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
output_path = tmp_path / "output.nc"
args = [
orography_path,
landmask_path,
sites_path,
"--all-methods",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /neighbour_selection_method_name - ['units']
improver_tests/acceptance/test_neighbour_finding.py:153: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --all-methods --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_global_global0/output.nc
______________________ test_alternative_coordinate_system ______________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python
tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_alternative_coordinate_sy0')
def test_alternative_coordinate_system(tmp_path):
"""Test use of an alternative coordinate system"""
kgo_dir = acc.kgo_root() / "neighbour-finding"
# same KGO as test_nearest global domain
kgo_path = kgo_dir / "outputs/nearest_global_kgo.nc"
sites_path = kgo_dir / "inputs/LAEA_grid_sites.json"
orography_path = kgo_dir / "inputs/global_orography.nc"
landmask_path = kgo_dir / "inputs/global_landmask.nc"
output_path = tmp_path / "output.nc"
coord_opts = {
"central_latitude": 54.9,
"central_longitude": -2.5,
"false_easting": 0.0,
"false_northing": 0.0,
"globe": {"semimajor_axis": 6378137.0, "semiminor_axis": 6356752.314140356},
}
coord_opts_json = json.dumps(coord_opts)
args = [
orography_path,
landmask_path,
sites_path,
"--site-coordinate-system",
"LambertAzimuthalEqualArea",
"--site-coordinate-options",
coord_opts_json,
"--site-x-coordinate",
"projection_x_coordinate",
"--site-y-coordinate",
"projection_y_coordinate",
"--output",
output_path,
]
run_cli(args)
> acc.compare(output_path, kgo_path)
E AssertionError: different attributes of /neighbour_selection_method_name - ['units']
improver_tests/acceptance/test_neighbour_finding.py:189: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/LAEA_grid_sites.json --site-coordinate-system LambertAzimuthalEqualArea --site-coordinate-options '{"central_latitude": 54.9, "central_longitude": -2.5, "false_easting": 0.0, "false_northing": 0.0, "globe": {"semimajor_axis": 6378137.0, "semiminor_axis": 6356752.314140356}}' --site-x-coordinate projection_x_coordinate --site-y-coordinate projection_y_coordinate --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_alternative_coordinate_sy0/output.nc
=========================== short test summary info ============================
FAILED improver_tests/acceptance/test_create_grid_with_halo.py::test_basic - ...
FAILED improver_tests/acceptance/test_apply_emos_coefficients.py::test_normal_point_by_point_sites
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest[global-global]
FAILED improver_tests/acceptance/test_create_grid_with_halo.py::test_halo_size
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_invalid_site
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_coord_beyond_bounds
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_unset_wmo_ids_with_unique_ids
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land[global-global]
FAILED improver_tests/acceptance/test_estimate_emos_coefficients.py::test_normal_point_by_point_default_initial_guess_sites
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_minimum_dz[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_all_methods[uk-ukvx]
FAILED improver_tests/acceptance/test_spot_extract.py::test_nearest_uk[all_methods_uk.nc-extra_args0-nearest_uk_temperatures.nc]
FAILED improver_tests/acceptance/test_spot_extract.py::test_nearest_uk[all_methods_uk.nc-extra_args1-mindz_uk_temperatures.nc]
FAILED improver_tests/acceptance/test_spot_extract.py::test_lapse_rate_adjusted_uk
FAILED improver_tests/acceptance/test_spot_extract.py::test_nearest_uk[all_methods_uk_unique_ids.nc-extra_args2-nearest_uk_temperatures_unique_ids.nc]
FAILED improver_tests/acceptance/test_spot_extract.py::test_new_spot_title - ...
FAILED improver_tests/acceptance/test_spot_extract.py::test_lapse_rate_mismatch
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_percentile_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_deterministic
FAILED improver_tests/acceptance/test_extract.py::test_spot - AssertionError:...
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_thresholded_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_deterministic_quiet
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_percentile_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_constraints
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_realization_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_thresholded_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_realization_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_from_threshold_with_realizations
FAILED improver_tests/acceptance/test_estimate_emos_coefficients.py::test_normal_point_by_point_sites
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_minimum_dz[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land_minimum_dz[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land_minimum_dz[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_all_methods[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_alternative_coordinate_system
= 36 failed, 305 passed, 4 skipped, 2 xfailed, 492 warnings in 137.60s (0:02:17) = |
dmentipl
added
the
MO review required
|
Thanks @dmentipl. This looks like good progress on support for Iris 3 and the changes needed look a lot less difficult than I was worried might be needed. I think it'd be good to add tests/support for Python 3.8, but tackle that in a separate follow up PR to this Iris 3 work. I expect that the main constraint there will be library builds available on conda-forge rather than any breaking changes in the Python language. I think the main outstanding questions are:
|
|
I have added a Python 3.8 environment. It is passing the unit tests. It fails the same acceptance tests as the Python 3.7 environment. The changes from the 3.7 env (other than Python version) are:
|
|
Hard changeover to Iris 3 - in principle fine, as so far it looks like migrating to it is mandatory this year. wmo_id units - what are the changes exactly? You are right that units just don't make sense for this, and as far as I can tell they are optional in CF for this kind of variable anyway. scipy version - we'd expect to move to ~1.6.2 with Iris 3 at the moment? |
That's good to know.
For '/wmo_id', and some others, the desired attribute dictionary is {'units': '1', 'long_name': 'wmo_id'}whereas the actual is {'long_name': 'wmo_id'}So it's just missing the units. I'm not sure why this is the case.
I've updated the scipy version in both environments. It leads to a few (8) more acceptance test failures (as @tjtg said above). |
|
I've now removed nimrod patching. Unit tests still pass, and the acceptance test failures are the same as prior. |
4 tasks
|
@dmentipl I have (scandalously) force pushed over your branch after rebasing and making some changes. The changes I have made are as follows:
I've run all the tests with both iris3 environments and everything passes as expected, as well as with our local environment. I still need to do a bit more testing on the HPC before this can all be merged at our end, but hopefully none of the changes I've made cause you any problems. I've promoted this PR to ready for review. Could you check that what I've done is okay for you guys. I'll finish up the HPC changes and get someone to look at this from our end as well. Then when it gets merged we will finally leave iris 2 behind for good. |
4 tasks
|
Thanks for making those changes @bayliffe 👍 They look good, in principle, from my perspective. However, even though the tests are passing in GitHub actions, I'm getting four unit test failures locally (using the conda environment files) in
Yes, I've got access, and I'm in the process of testing these. Here are the details of the unit-test failures referenced above (for the Python 3.8 env; the errors are the same for the Python 3.7 env): ================================================ FAILURES =================================================
___________________________ Test_pad_and_unpad_cube.test_single_point_on_corner ___________________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_pad_and_unpad_cube testMethod=test_single_point_on_corner>
def test_single_point_on_corner(self):
"""Test behaviour for a single non-zero grid cell on the corner."""
expected = np.array(
[
[
[0.0, 0.4, 1.0, 1.0, 1.0],
[0.4, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[0.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
],
]
)
self.cube.data[0, 0] = 0
# Point is right on the corner.
percentiles = np.array([10, 50, 90])
kernel = np.array([[0.0, 1.0, 0.0], [1.0, 1.0, 1.0], [0.0, 1.0, 0.0]])
result = GeneratePercentilesFromACircularNeighbourhood(
percentiles=percentiles
).pad_and_unpad_cube(self.cube, kernel)
> self.assertArrayAlmostEqual(result.data, expected)
improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:347:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_pad_and_unpad_cube testMethod=test_single_point_on_corner>
a = array([[[0. , 0.4, 1. , 1. , 1. ],
[0.4, 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ],
[1. ,... [1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[0. , 0.4, 1. , 1. , 1. ],
[0.4, 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ],
[1. ,..., 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ]]])
decimal = 6
def assertArrayAlmostEqual(self, a, b, decimal=6):
> np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E AssertionError:
E Arrays are not almost equal to 6 decimals
E
E Mismatched elements: 1 / 75 (1.33%)
E Max absolute difference: 1.
E Max relative difference: 0.00000001
E x: array([[[0. , 0.4, 1. , 1. , 1. ],
E [0.4, 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 1. , 1. ],...
E y: array([[[0. , 0.4, 1. , 1. , 1. ],
E [0.4, 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 1. , 1. ],...
../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
_______________________ Test_run.test_number_of_percentiles_equals_number_of_points _______________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_percentiles_equals_number_of_points>
def test_number_of_percentiles_equals_number_of_points(self):
"""Test when the number of percentiles is equal to the number of points
used to construct the percentiles."""
expected = np.array(
[
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.2, 0.2, 0.2, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.4, 0.4, 0.4, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.6, 0.6, 0.6, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.8, 0.8, 0.8, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
]
)
data = np.ones((7, 7), dtype=np.float32)
data[3, 3] = 0
cube = set_up_variable_cube(data, spatial_grid="equalarea",)
percentiles = np.array([5, 10, 15, 20, 25])
radius = 2000.0
result = GeneratePercentilesFromACircularNeighbourhood(
percentiles=percentiles
).run(cube, radius)
> self.assertArrayAlmostEqual(result.data, expected)
improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:715:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_percentiles_equals_number_of_points>
a = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 0.2,..., 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 0.2,..., 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ]]])
decimal = 6
def assertArrayAlmostEqual(self, a, b, decimal=6):
> np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E AssertionError:
E Arrays are not almost equal to 6 decimals
E
E Mismatched elements: 1 / 245 (0.408%)
E Max absolute difference: 0.6
E Max relative difference: 1.5
E x: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 0.2, 1. , 1. , 1. ],...
E y: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 0.2, 1. , 1. , 1. ],...
../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
______________________ Test_run.test_number_of_points_half_of_number_of_percentiles _______________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_points_half_of_number_of_percentiles>
def test_number_of_points_half_of_number_of_percentiles(self):
"""Test when the number of points is half the number of percentiles."""
expected = np.array(
[
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.1, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.1, 0.1, 0.1, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.1, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.2, 0.2, 0.2, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.3, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.3, 0.3, 0.3, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.3, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.4, 0.4, 0.4, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.5, 0.5, 0.5, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.6, 0.6, 0.6, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.7, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.7, 0.7, 0.7, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.7, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.8, 0.8, 0.8, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.9, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.9, 0.9, 0.9, 1.0, 1.0],
[1.0, 1.0, 1.0, 0.9, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
],
]
)
data = np.ones((7, 7), dtype=np.float32)
data[3, 3] = 0
cube = set_up_variable_cube(data, spatial_grid="equalarea",)
percentiles = np.array([2.5, 5, 7.5, 10, 12.5, 15, 17.5, 20, 22.5, 25])
radius = 2000.0
result = GeneratePercentilesFromACircularNeighbourhood(
percentiles=percentiles
).run(cube, radius)
> self.assertArrayAlmostEqual(result.data, expected)
improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:824:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_points_half_of_number_of_percentiles>
a = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 0.1,..., 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 0.1,..., 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 1. , 1. , 1. , 1. , 1. ]]])
decimal = 6
def assertArrayAlmostEqual(self, a, b, decimal=6):
> np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E AssertionError:
E Arrays are not almost equal to 6 decimals
E
E Mismatched elements: 9 / 490 (1.84%)
E Max absolute difference: 0.8
E Max relative difference: 4.
E x: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 0.1, 1. , 1. , 1. ],...
E y: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 1. , 0.1, 1. , 1. , 1. ],...
../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
_______________________________ Test_run.test_single_point_low_percentiles ________________________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_single_point_low_percentiles>
def test_single_point_low_percentiles(self):
"""Test behaviour with low percentiles."""
expected = np.array(
[
[
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.2, 1.0, 1.0],
[1.0, 0.2, 0.2, 0.2, 1.0],
[1.0, 1.0, 0.2, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.4, 1.0, 1.0],
[1.0, 0.4, 0.4, 0.4, 1.0],
[1.0, 1.0, 0.4, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
],
[
[1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 0.8, 1.0, 1.0],
[1.0, 0.8, 0.8, 0.8, 1.0],
[1.0, 1.0, 0.8, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 1.0],
],
]
)
self.cube.data[2, 2] = 0
percentiles = np.array([5, 10, 20])
radius = 2000.0
result = GeneratePercentilesFromACircularNeighbourhood(
percentiles=percentiles
).run(self.cube, radius)
> self.assertArrayAlmostEqual(result.data, expected)
improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:613:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_single_point_low_percentiles>
a = array([[[1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 0.2, 1. , 1. ],
[1. , 0.2, 0.2, 0.2, 1. ],
[1. ,... [1. , 0.8, 0.8, 0.8, 1. ],
[1. , 1. , 0.8, 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. ],
[1. , 1. , 0.2, 1. , 1. ],
[1. , 0.2, 0.2, 0.2, 1. ],
[1. ,..., 1. , 1. ],
[1. , 0.8, 0.8, 0.8, 1. ],
[1. , 1. , 0.8, 1. , 1. ],
[1. , 1. , 1. , 1. , 1. ]]])
decimal = 6
def assertArrayAlmostEqual(self, a, b, decimal=6):
> np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E AssertionError:
E Arrays are not almost equal to 6 decimals
E
E Mismatched elements: 2 / 75 (2.67%)
E Max absolute difference: 0.6
E Max relative difference: 1.5
E x: array([[[1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 0.2, 1. , 1. ],
E [1. , 0.2, 0.2, 0.2, 1. ],...
E y: array([[[1. , 1. , 1. , 1. , 1. ],
E [1. , 1. , 0.2, 1. , 1. ],
E [1. , 0.2, 0.2, 0.2, 1. ],...
../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
========================================= short test summary info =========================================
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_pad_and_unpad_cube::test_single_point_on_corner
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_number_of_percentiles_equals_number_of_points
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_number_of_points_half_of_number_of_percentiles
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_single_point_low_percentiles
================= 4 failed, 2428 passed, 319 skipped, 1 xfailed, 3568 warnings in 48.88s ================== |
|
I've determined that the errors in the unit tests reported above are due to the version of NumPy. In my local testing environment I am using We can pin numpy to I have also checked that the acceptance tests pass with the new data (in diff --git a/home/dmentipl/tmp/improver_py38_iris30-github.txt b/home/dmentipl/tmp/improver_py38_iris30-local.txt
index 76989d40..53f60e2c 100644
--- a/home/dmentipl/tmp/improver_py38_iris30-github.txt
+++ b/home/dmentipl/tmp/improver_py38_iris30-local.txt
@@ -1,4 +1,4 @@
-# packages in environment at /usr/share/miniconda/envs/improver_py38_iris30:
+# packages in environment at /home/dmentipl/conda/envs/improver_py38_iris30:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
@@ -7,7 +7,7 @@ alabaster 0.7.12 py_0 conda-forge
alsa-lib 1.2.3 h516909a_0 conda-forge
antlr-python-runtime 4.7.2 py38h578d9bd_1002 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
-astroid 2.5.8 py38h578d9bd_0 conda-forge
+astroid 2.6.0 py38h578d9bd_0 conda-forge
attrs 19.3.0 py_0 conda-forge
babel 2.9.1 pyh44b312d_0 conda-forge
bandit 1.7.0 py38h578d9bd_0 conda-forge
@@ -34,10 +34,10 @@ cryptography 3.4.7 py38ha5dfef3_0 conda-forge
curl 7.77.0 hea6ffbf_0 conda-forge
cycler 0.10.0 py_2 conda-forge
cytoolz 0.11.0 py38h497a2fe_3 conda-forge
-dask 2021.6.1 pyhd8ed1ab_0 conda-forge
-dask-core 2021.6.1 pyhd8ed1ab_0 conda-forge
-dbus 1.13.18 hb2f20db_0
-distributed 2021.6.1 py38h578d9bd_0 conda-forge
+dask 2021.6.2 pyhd8ed1ab_0 conda-forge
+dask-core 2021.6.2 pyhd8ed1ab_0 conda-forge
+dbus 1.13.6 h48d8840_2 conda-forge
+distributed 2021.6.2 py38h578d9bd_0 conda-forge
docutils 0.17.1 py38h578d9bd_0 conda-forge
dparse 0.5.1 py_0 conda-forge
execnet 1.9.0 pyhd8ed1ab_0 conda-forge
@@ -49,14 +49,14 @@ fontconfig 2.13.1 hba837de_1005 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
fsspec 2021.6.0 pyhd8ed1ab_0 conda-forge
geos 3.9.1 h9c3ff4c_2 conda-forge
-gettext 0.21.0 hf68c758_0
+gettext 0.19.8.1 h0b5b191_1005 conda-forge
gitdb 4.0.7 pyhd8ed1ab_0 conda-forge
gitpython 3.1.18 pyhd8ed1ab_0 conda-forge
glib 2.68.3 h9c3ff4c_0 conda-forge
glib-tools 2.68.3 h9c3ff4c_0 conda-forge
gmp 6.2.1 h58526e2_0 conda-forge
-gnutls 3.6.15 he1e5248_0
-graphite2 1.3.14 h23475e2_0
+gnutls 3.6.13 h85f3911_1 conda-forge
+graphite2 1.3.13 he1b5a44_1001 conda-forge
gst-plugins-base 1.18.4 hf529b03_2 conda-forge
gstreamer 1.18.4 h76c114f_2 conda-forge
harfbuzz 2.8.1 h83ec7ef_0 conda-forge
@@ -90,7 +90,7 @@ libcblas 3.9.0 9_openblas conda-forge
libclang 11.1.0 default_ha53f305_1 conda-forge
libcurl 7.77.0 h2574ce0_0 conda-forge
libdeflate 1.7 h7f98852_5 conda-forge
-libedit 3.1.20210216 h27cfd23_1
+libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libffi 3.3 h58526e2_2 conda-forge
@@ -100,14 +100,13 @@ libgfortran5 9.3.0 hff62375_19 conda-forge
libglib 2.68.3 h3e27bee_0 conda-forge
libgomp 9.3.0 h2828fa1_19 conda-forge
libiconv 1.16 h516909a_0 conda-forge
-libidn2 2.3.1 h7f98852_0 conda-forge
liblapack 3.9.0 9_openblas conda-forge
liblapacke 3.9.0 9_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libllvm11 11.1.0 hf817b99_2 conda-forge
libnetcdf 4.8.0 nompi_hcd642e3_103 conda-forge
libnghttp2 1.43.0 h812cca2_0 conda-forge
-libogg 1.3.5 h27cfd23_1
+libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.15 pthreads_h8fe5266_1 conda-forge
libopencv 4.5.2 py38hcdf9bf1_0 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
@@ -116,18 +115,16 @@ libpq 13.3 hd57d9b9_0 conda-forge
libprotobuf 3.15.8 h780b84a_0 conda-forge
libssh2 1.9.0 ha56f1ee_6 conda-forge
libstdcxx-ng 9.3.0 h6de172a_19 conda-forge
-libtasn1 4.16.0 h27cfd23_0
libtiff 4.3.0 hf544144_1 conda-forge
-libunistring 0.9.10 h14c3975_0 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libvorbis 1.3.7 he1b5a44_0 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
-libxcb 1.14 h7b6447c_0
+libxcb 1.13 h7f98852_1003 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.12 h72842e0_0 conda-forge
libzip 1.8.0 h4de3113_0 conda-forge
llvmlite 0.36.0 py38h4630a5e_0 conda-forge
-locket 0.2.1 py38h06a4308_1
+locket 0.2.0 py_2 conda-forge
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
markupsafe 2.0.1 py38h497a2fe_0 conda-forge
matplotlib-base 3.4.2 py38hcc49a3a_0 conda-forge
@@ -135,17 +132,17 @@ mccabe 0.6.1 py_1 conda-forge
mock 4.0.3 py38h578d9bd_1 conda-forge
more-itertools 8.8.0 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.2 py38h1fd1430_1 conda-forge
-mypy 0.902 py38h497a2fe_0 conda-forge
+mypy 0.910 py38h497a2fe_0 conda-forge
mypy_extensions 0.4.3 py38h578d9bd_3 conda-forge
mysql-common 8.0.25 ha770c72_2 conda-forge
mysql-libs 8.0.25 hfa10184_2 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
netcdf4 1.5.7 nompi_py38h5e9db54_100 conda-forge
-nettle 3.7.3 hbbd107a_1
+nettle 3.6 he412f7d_0 conda-forge
nspr 4.30 h9c3ff4c_0 conda-forge
nss 3.67 hb5efdd6_0 conda-forge
numba 0.53.1 py38h8b71fd7_1 conda-forge
-numpy 1.20.3 py38h9894fe3_1 conda-forge
+numpy 1.21.0 py38h9894fe3_0 conda-forge
od 1.0 py_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencv 4.5.2 py38h578d9bd_0 conda-forge
@@ -153,7 +150,7 @@ openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1k h7f98852_0 conda-forge
packaging 20.9 pyh44b312d_0 conda-forge
-pandas 1.2.4 py38h1abd341_0 conda-forge
+pandas 1.2.5 py38h1abd341_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pathspec 0.8.1 pyhd3deb0d_0 conda-forge
patsy 0.5.1 py_0 conda-forge
@@ -165,6 +162,7 @@ pixman 0.40.0 h36c2ea0_0 conda-forge
pluggy 0.13.1 py38h578d9bd_4 conda-forge
proj 7.2.0 h277dcde_2 conda-forge
psutil 5.8.0 py38h497a2fe_1 conda-forge
+pthread-stubs 0.4 h36c2ea0_1001 conda-forge
py 1.10.0 pyhd3deb0d_0 conda-forge
py-opencv 4.5.2 py38hd0cf306_0 conda-forge
pycodestyle 2.7.0 pyhd8ed1ab_0 conda-forge
@@ -196,7 +194,7 @@ regex 2021.4.4 py38h497a2fe_0 conda-forge
requests 2.25.1 pyhd3deb0d_0 conda-forge
safety 1.10.3 pyhd8ed1ab_0 conda-forge
scipy 1.6.3 py38h7b17777_0 conda-forge
-setuptools 52.0.0 py38h06a4308_0
+setuptools 49.6.0 py38h578d9bd_3 conda-forge
shapely 1.7.1 py38haeee4fe_5 conda-forge
sigtools 2.0.2 py38h578d9bd_3 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
@@ -231,6 +229,8 @@ xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
+xorg-libxau 1.0.9 h14c3975_0 conda-forge
+xorg-libxdmcp 1.1.3 h516909a_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
|
|
Further digging shows in and the result (note the misplaced |
|
@dmentipl thanks for looking into this. Let's pin the version of numpy in the envs for now and move this work forward. If you could then create a ticket in this repo explaining what you found (or linking to the comments above), we can address this issue in slower time before we are compelled to move to numpy 1.21.0. Sound okay? |
This was referenced Jun 25, 2021
|
@bayliffe no worries, thanks for all your assistance. I've pinned numpy to Also, I've pinned cf-units to So, from our (BoM) perspective this PR is ready to merge pending review, and any outstanding Met Office requirements. |
|
We (Mostly @MoseleyS) have found a problem between Iris adds an extra coord, it used to be This then throws the error This is now being looked into so it can be replicated with acceptance tests. |
|
@tjtg @dmentipl Would it be possible to update improver branch iris3 to include changes in master (in particular I am interested in including #1528)? We are using improver/iris3 branch in raincal CIs but need #1528 - thanks. cc @benowen-bom |
|
@lucyleeow I've rebased against master and force pushed. |
|
I have pinned cftime to < 1.5 as compatibility with cf-units 2.1.5 breaks. |
Merged
No longer available in Iris 3.0. Also removed "strict" keyword arg to extract.
Requires specification of the interpolation order, which was previously always 0 (nearest neighbour).
Iris 3 includes functionality that equalises attributes in the way that we require, so the maintenance of our own method is no longer required.
Version 3.0.0 was recently released and using it in IMPROVER results in ~40 unit test failures.
Plus, replace "==" with "=" in pinning cf-units.
#1) * Minor edits to support use of cftime in datetime_to_iris_time function. * Adjust type annotations Co-authored-by: Daniel Mentiplay <[email protected]>
|
I have brought this branch up to date and I think it's ready to approve/merge now:
|
zfan001
approved these changes
Aug 30, 2021
MoseleyS
pushed a commit
to MoseleyS/improver
that referenced
this pull request
Aug 22, 2024
* Remove reference to LimitedAttributeDict This was used for type annotations and relied on a "private" module. * Add conda env for Iris 3.0 * Remove extract_strict No longer available in Iris 3.0. Also removed "strict" keyword arg to extract. * Add Iris 3.0 conda env to testing * Skip failing test for now * Mark test as xfail rather than skip * Drop Iris 2.* envs from the test matrix * Update setup.cfg and docs to Iris 3.0 * Add Python 3.8 environment * Use function instead of method in test * Update scipy version and unpin cftime * Remove nimrod patching for Iris * Remove unused imports * Update pysteps version. Requires specification of the interpolation order, which was previously always 0 (nearest neighbour). * Update checksums. * Remove IMPROVER equalise attributes function. Iris 3 includes functionality that equalises attributes in the way that we require, so the maintenance of our own method is no longer required. * Remove Iris 2 environments with which IMPROVER is no longer backwards compatible. * Add pytest-xdist to environments. * Pin numpy version to 1.20.* * Pin cf-units to 2.1.5 Version 3.0.0 was recently released and using it in IMPROVER results in ~40 unit test failures. * Allow numpy versions < 1.21 in conda envs Plus, replace "==" with "=" in pinning cf-units. * Remove Iris 2 env * Adjust cftime and cf-units versions in envs * Make Python 3.8 env consistent with 3.7 env * Minor edits to support use of cftime in datetime_to_iris_time function (#1) * Minor edits to support use of cftime in datetime_to_iris_time function. * Adjust type annotations Co-authored-by: Daniel Mentiplay <[email protected]> * Spot extract KGOs metoppv#1521 plus units attribute Co-authored-by: benjamin.ayliffe <[email protected]> Co-authored-by: gavinevans <[email protected]> Co-authored-by: Tom Gale <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This (draft) PR adds support to IMPROVER for Iris 3.0 at the expense of supporting Iris 2.*. I have marked this PR as a draft so it can be used as a place for further discussion.
Here is a summary of the main changes:
CubeList.extract_strictand use ofstrictkeyword argument toCubeList.extractand replaced them withCubeList.extract_cube(for more, see here in the Iris docs changelog)from iris._cube_coord_common import LimitedAttributeDictwhich was introduced for type annotationsenvironment_py37_iris30.ymlwhich has the same environment asenvironment_py37_iris24.ymlexcept for replacing Iris 2.4 for 3.0setup.cfgFurther issues to be addressed:
environment_py37_iris24.ymlwith Iris 3.0 and Python 3.8)Addresses #1437, #1517
Testing:
Currently the unit tests are passing in the py37_iris30 environment. However, I marked one test
Test_lat_lon_determineinimprover_tests/utilities/test_spatial.pyas xfail. I did some digging but could not find why it was failing.