Testing/CI/Release For Embedded Targets

[posborne]

Problem

Embedded targets typically do not target the same architectures that are readily available on CI services like Travis and developers machines, yet we would still like for CI to be able to build our software for alternative architectures and, ideally, run tests on that architecture.

As an example, let's consider another project I am a maintainer on, nix-rust/nix, which although not strictly focused on embedded is likely to be needed on many embedded systems (it is a dependency of mio and many other crates as it provides a set of safer APIs on top of libc which may not be present in std). For this project, we want to do the following:

• Ensure the software builds correctly
• Ensure the software built for passes all unit tests
• (For Rust Applications) Generate working debug/release binaries

Matrixed with these additional things for each of the above:

• With each major feature combination
• On each supported target
• On each support version of Rust

Project Case Studies

Currently, there are several projects that implement their own solutions to this problem (to varying degrees of success) and a few projects which exist to help aid developers who are seeking to build/test for several different platforms.

rust-lang/libc

The libc crate is built for and runs tests against a number of different targets including several which are not yet officially supported. The libc crate contains a CI Directory which provides an overview of the strategy it uses for doing cross-build/testing.

This boils down to the following (ignoring platforms like Windows/OSX that are not really relevant for embedded):

• Triples are specified using the TARGET variable and the desired rust version is specified with the rust variable in the travis matrix. Linux is used for the host OS. E.g.

    - os: linux
      env: TARGET=arm-unknown-linux-gnueabihf
      rust: stable
    - os: linux
      env: TARGET=x86_64-unknown-openbsd QEMU=openbsd.qcow2
      rust: stable
      script: sh ci/run-docker.sh $TARGET

• The ci/run-docker.sh script will do the following for $TARGET:
  • Runs docker build to create a docker image using the Dockerfile in ci/docker/$TARGET. E.g. https://github.com/rust-lang/libc/blob/master/ci/docker/arm-unknown-linux-gnueabihf/Dockerfile. The docker images contain all non-rust dependencies that are required to build and test for $TARGET. Typically, this is gcc/libc (for the libc crate) and qemu-user.
  • Runs docker run to launch a new container from the previously built image and executes the run.sh script
  • With the run command, the rust sysroot (-v rustc --print sysroot:/rust:ro) is shared with the container at /rust
  • If the QEMU environment variable is specified, the QEMU image is fetched and and a bunch of rigamarole and setup is performed in order to set things up. Finally, QEMU is run and the output is parsed to determine success. The only emulated machine currently is x86_64.
  • QEMU- is automatically selected for specific targets (see https://github.com/rust-lang/libc/blob/master/ci/run.sh#L107) and QEMU is just used for running the libc-test binary (cross compilation of this binary for the target is done on the host.

Things work well, but there is a moderate amount of complexity. Effort that is done to improve testing for the libc crate do not directly help out other projects which might want the same enhancements.

Major contributors here have been @alexchrichton, @japaric, @semarie

nix-rust/nix

This nix-rust/nix crate is based off of the work done in libc in an earlier version and has diverged some since then. This work was originally done by yours truly.

• Travis is used to perform testing for the various *nix platforms that are supported.
• Cross target builds are specified as follows:

    - os: linux
      env: TARGET=aarch64-unknown-linux-gnu DOCKER_IMAGE=posborne/rust-cross:arm
      rust: 1.7.0
      sudo: true
    - os: linux
      env: TARGET=arm-unknown-linux-gnueabihf DOCKER_IMAGE=posborne/rust-cross:arm
      rust: 1.7.0
      sudo: true
    - os: linux
      env: TARGET=mips-unknown-linux-gnu DOCKER_IMAGE=posborne/rust-cross:mips
      rust: 1.7.0
      sudo: true
    - os: linux

• At present, it is assumed that containers that will be used are published. No images are built as part of the travis build process itself
• Rust and the sysroot for targets supported by the docker image are built into the target image itself rather than being mounted into the container as is the case with libc.
• CI executes the run-docker.sh script which for cross-targets will in turn call run-docker.sh
• This script will run (and pull) the specified docker container from dockerhub, executing the run.sh script in the container.
• Cross compilation will be performed and, similar to libc, based on the Target we will either execute the tests directly or run them using QEMU.
• Since nix has multiple test files (as will most projects other than libc), there are some hacks in order to attempt to figure out what those are so they can be run: https://github.com/nix-rust/nix/blob/master/ci/run.sh#L66

This work was done by @posborne based on libc.

japaric/smoke

I haven't looked at this one much but @japaric has some relevant experience. Appears to use some combination of Docker/QEMU for build/testing. Dockerfile appears to be monolithic.

Others?

Looking for feedback on other projects that are doing a non-trivial amount with doing cross-build/test within the Rust ecosystem. MCU targets would be appreciated in addition to the above projects which focus on targets with an OS.

Ecosystem Projects

rust-embedded/docker-rust-cross

The goal of this repository was to provide the Docker images (a common theme for this work) in order to perform cross-compilation and cross-test for non-host architectures. To date, I have not been diligent about keeping these images up-to-date with each Rust release.

The goal is to have a common repository of Docker images that can be reused across projects like libc/nix/others so each project doesn't need to go in the weeds on that front.

@posborne is the maintainer of this project.

japaric/rust-everywhere

Rust-everywhere seeks to make it easier to cross-compile crates for other targets and publish binaries. It does not appear to help out with running tests on foreign architectures, although this is something that is likely to be desirable for projects targeting other architectures.

Final Note

For many libraries that do not do FFI (especially libc/kernel FFI), the importance of actually running tests on the target architecture is probably not as strong. Within embedded, however, there seems to be a much greater chance that using APIs that may not already have nice interfaces is increased. As such, I feel this forum is still an appropriate place to discuss how we want to handle this problem moving forward.

[thejpster]

I'm building with Xargo on Travis for Cortex-M4 but I don't have a strategy for test as yet (so maybe that counts as 'trivial', I don't know). At work, we would generally hook up target hardware to our CI system to flash binaries and run tests on target, but this is a personal project for now.

[japaric]

I have been working in this space 😄.

I really liked libc idea of one docker container per target. It makes builds reproducible, you can easily run the test suite for any target locally (if you have Docker and are running Linux) and its Dockerfiles let you pick which glibc version you want to compile against (through the Ubuntu image tag/version).

While building smoke (a repository that's meant to test the nightly releases of libstd against as many targets as possible (though it's not exactly running on a nightly basis right now)), I figured out how to transparently run cross compiled programs under QEMU. By transparent, I meant that cargo test --target arm-unknown-linux-gnueabi will cross compile the Rust program on the host and then run the cross compiled test runner inside QEMU without having to mention qemu at all in the actual command.

I'm also the author of rust-everywhere which is a CI "template" that produces binary releases of a crate for tier 1 targets (Linux, OSX and Windows) and uploads them to GitHub releases.

So, lately I have been trying to merge these three concepts: one Docker image per target, transparent cross testing using QEMU and deploys (binary releases). The result so far is trust. Which is, right now, also a CI "template" that uses Docker and QEMU to test and make deploys for 20+ different targets (including the new thumb targets; these can't be cargo tested (no libtest) but they do have deploys)

Trust is already functional. I have been using it as a CI template in my latest embedded projects to test my crates for the thumb targets. But it's missing a good UI to be massified. Something that went wrong with rust-everywhere is that it didn't have an upgrade mechanism and as a result the CI files in the rust-everywhere repo went through many improvements but the users were never notified of these and if they did notice, they didn't have any way to easily upgrade their CI files which also had already diverged from the original CI template they used.

So, yeah. I wouldn't anyone to use trust until there's an upgrade mechanism in place.

---

For many libraries that do not do FFI (especially libc/kernel FFI), the importance of actually running tests on the target architecture is probably not as strong.

Pure Rust libraries that (want to) optimize their implementations on some architectures using assembly should also test on different targets. Libraries in this category are compiler-builtins and m.

If you are using repr(C) or mem::transmuteing stuff, I'd also recommend testing your crate on a big endian target and also on a target with 32-bit pointers just to make sure you are making any assumptions about the layout of structs or the size of pointers.

[japaric]

@jamesmunns wrote an excellent blog post about the different ways to do CI testing in the embedded world. The approaches listed there are:

CI Build

Basically run xargo build --target $T (or cross build) on Travis to make sure the crate doesn't stop compiling / linking.

I think this method is well supported today thanks to cross / xargo.

Non-Host Testing

Basically cargo test.

You have structure your embedded crate correctly so that unit tests can be run on the host. This is mainly for testing logic that doesn't involve target device features / hardware as those are not present on Travis builders / your laptop. This approach is not "high fidelity" as there are some differences between the target device and the Travis builder: for example, the size of usize, *const T values.

Host Testing

Basically xargo test --target $T --no-run and then copy and run the tests on the target device.

As you may know the test crate depends on std so that crate can't be used to test no_std crates. But I have created an alternative test crate, the utest framework, that can be used to compile unit tests and run them on real hardware. I haven't used much utest to run tests on real hardware so I don't know if it's good enough to test device functionality like I2C.

Simulated Host Testing

Basically cross test --target $T; this runs the unit tests under QEMU.

Again test doesn't support this but the utest framework does. I have a deploy of this method in the compiler-builtins crate and it has catched several bug in the implementations of intrinsics after we starting running unit tests on the thumbv*m targets.

Do note that QEMU doesn't emulate device hardware like I2C and PWM so you can't test that functionality using this method.

Hardware In the Loop (HIL)

This would be some higher level kind of integration testing where a PC sends some input to the target device, the target device responds with some output and the PC verifies that the output is indeed correct.

AFAIK, no solution for this exists in the ecosystem.

---

I think that both CI builds and non-host testing are well supported today and can be easily be implemented using Travis CI.

The utest framework can be used for host testing and simulated host testing but is it in good shape enough? Could it be improved further? How does it compare to other implementations used in the C/C++ world? I'd like more people to give it a try, specially in the host testing deparment.

Does anyone has any idea of how a HIL framework would look like for Rust? Or does anyone know of a well established HIL framework that we could port to Rust?

[thejpster]

Most of my work products achieve HIL testing by exposing the functional API at each level of the stack through a command line harness. This can compile for either host and embedded target. You can drive the interface manually or with an automated tool.

[jcsoo]

I recently released Bobbin CLI, which includes a simple text-based serial console based test runner. I experimented with a full-fledged two-way COBS encoded TLV-style protocol, but eventually decided that I wanted something much, much simpler.

$ bobbin test
   Compiling frdm-k64f v0.1.0 (file:///home/bobbin/bobbin-boards/frdm-k64f)
    Finished dev [optimized + debuginfo] target(s) in 0.61 secs
   text	   data	    bss	    dec	    hex	filename
   6252	    428	    408	   7088	   1bb0	target/thumbv7em-none-eabihf/debug/frdm-k64f
     Loading target/thumbv7em-none-eabihf/debug/frdm-k64f
    Complete Successfully flashed device
      Loader Load Complete
     Console Opening Console
[start] Running tests for frdm-k64f
[pass] Test 0
[pass] Test 1
[pass] Test 2
[pass] Test 3
[pass] Test 4
[done] All tests passed
$

bobbin test recognizes [start], [pass] and [done] tags, exiting with return code 0. It also recognizes [fail], [exception], and [panic] tags, which will cause it to exit with return codes 1, 2 or 3. All other output is ignored.

The test runner will exit with return code 1 if there is a delay of more than 5 seconds between lines or 15 seconds to complete the entire test. In the future these timeouts will be configurable.

This system doesn't currently handle tests that are intentionally supposed to trap or panic, but I can think of a few ways to make it possible with some help from a watchdog timer.

Currently this is only implemented for devices with serial consoles, but it should be straightforward to make this work using SWO for targets and debuggers that support it.

[rubberduck203]

You have structure your embedded crate correctly so that unit tests can be run on the host

@japaric do you know of any examples of anyone doing this? I’m very interested in your work on utest, but I’m currently interested in how to organize a crate so that I can run whatever tests I can on host.

In C++ I essentially have 2 makefiles. One for host and one for target. Any idea of how to accomplish something similar with Rust?

[jamesmunns]

Closing this as part of 2024 triage.

As of now, the charter of the WG is to focus on foundational crates and functionality, rather than develop solutions for all use cases. However, as of today there are the following existing solutions, and folks finding this issue are suggested to contribute to one of the following projects:

• https://crates.io/crates/embedded-test
• https://docs.rs/defmt-test/latest/defmt_test/
• https://github.com/embassy-rs/teleprobe

If you think this was closed incorrectly, please leave a comment and we can revisit this decision in the next weekly WG meeting on Matrix.