Skip to content

Commit

Permalink
ci: install HDF5 and C-blosc explicitly for pytables issue on MacOS
Browse files Browse the repository at this point in the history
This commit tries to fix some flakiness of the test execution over MacOS-based instances over GitHub Actions.

It appears to be an issue of the `tables` dependency. From the logs of the CI: (https://github.com/bancaditalia/black-it/actions/runs/7121188681/job/19389914846?pr=78#step:5:62)

```
Fatal Python error: Illegal instruction

Current thread 0x000000011783c600 (most recent call first):
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/node.py", line 246 in __init__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/leaf.py", line 264 in __init__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/carray.py", line 200 in __init__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/earray.py", line 143 in __init__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/file.py", line 1345 in create_earray
  File "/Users/runner/work/black-it/black-it/black_it/utils/json_pandas_checkpointing.py", line 211 in save_calibrator_state
  File "/Users/runner/work/black-it/black-it/tests/test_utils/test_pandas_json_checkpointing.py", line 58 in test_save_and_load_calibrator_state
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/Users/runner/work/black-it/black-it/.tox/py310/bin/pytest", line 8 in <module>
```

A successful workaround seems to install HDF5 and C-blosc explicitly in the preliminary steps on CI for MacOS images.
This step was part of the installation instructions of PyTables v3.9.2: https://github.com/PyTables/PyTables/blob/v3.9.2/README.rst#installation
Moreover, as explained here: https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac, sometimes must be explicitly set the environment path variables where HDF5 ahd C-blosc are installed

For some reason this is not needed on Ubuntu and Windows runners.
  • Loading branch information
marcofavoritobi committed Dec 7, 2023
1 parent 9cb07d0 commit 0b65803
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 90 deletions.
46 changes: 0 additions & 46 deletions .github/workflows/docs.yml

This file was deleted.

39 changes: 0 additions & 39 deletions .github/workflows/lint.yml

This file was deleted.

19 changes: 14 additions & 5 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:

strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
os: [macos-11]
python-version: [3.8, 3.9, "3.10", "3.11"]

timeout-minutes: 30
Expand All @@ -37,14 +37,23 @@ jobs:
# env:
# ACTIONS_ALLOW_UNSECURE_COMMANDS: true
# run: ...
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-latest') }}
name: Unit tests and coverage (ubuntu-latest, macos-latest)
- if: matrix.os == 'macos-11'
name: Install dependencies (macos-11)
run: |
# for some reason this seems to be a necessary step for MacOS images, but not for Ubuntu and Windows
brew install hdf5
brew install c-blosc
# https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac
echo "HDF5_DIR=/usr/local/" >> $GITHUB_ENV
echo "BLOSC_DIR=/usr/local/" >> $GITHUB_ENV
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-11') }}
name: Unit tests and coverage (ubuntu-latest, macos-11)
run: |
pyver=`echo ${{ matrix.python-version }} | tr -d "."`
tox -e py${pyver} -- -m 'not e2e'
tox -e py${pyver}-nb
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-latest') }}
name: Notebook tests (ubuntu-latest, macos-latest)
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-11') }}
name: Notebook tests (ubuntu-latest, macos-11)
run: |
pyver=`echo ${{ matrix.python-version }} | tr -d "."`
tox -e py${pyver}-nb
Expand Down

0 comments on commit 0b65803

Please sign in to comment.