Run tests with miri

[swfsql]

It's possible to use miri to run some tests (cpu ones - so no cuda feature - and the ones that don't interact with python - so no numpy feature). Miri can be particularly useful to detect possible Undefined Behavior cases and memory leaks, although it can also have false positives (and false negatives - this depends on the tests coverage).

Setup

The miri readme mentions using cargo nextest so tests are able to run in parallel, and for memory leaks to be detected.

# install miri
rustup +nightly component add miri

# install cargo nextest (Linux x86_64)
curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin

# cargo clean (run once when `cargo miri` is not used)
cargo clean

# run all tests (note: very slow and should take hours, assuming there are no errors)
MIRIFLAGS='-Zmiri-strict-provenance -Zmiri-symbolic-alignment-check -Zmiri-retag-fields -Zmiri-disable-isolation -Zmiri-tree-borrows' \
cargo miri nextest run --test-threads 1 --no-default-features --features std,fast-alloc,cpu,nightly,safetensors

Note that miri is very slow and takes an hour to run eg. the longest transformer test. So an accompanying issue (#898) would be to reduce the test sizes to their minimum.

[swfsql]

Closing as I think the tests that got run on the linked pr indicated a robust behavior.
But feel free to ping or re-open in case anyone has an interest!