Skip to content

Spec Tests

Jump to bottom Edit New page
andychu edited this page Jun 9, 2024 · 43 revisions

Back to Contributing.

Spec Tests are written with the sh_spec.py framework. There are some comments at the top of that file.

They live in files like spec/smoke.test.sh, and can be run with shell functions test/spec.sh smoke (sorry they're in different dirs)

Quick Start

Follow these steps in Contributing:

$ build/deps.sh fetch 
$ build/deps.sh install-wedges-fast

Then

$ test/spec.sh smoke          # run a single file, which lives in spec/smoke.test.sh

$ test/spec-py.sh ysh-all     # all YSH spec tests in parallel
$ test/spec-py.sh osh-all     # all OSH spec tests in parallel - this may be flaky

Useful shortcuts:

$ test/spec.sh smoke --range 11-12  # just run two tests
$ test/spec.sh smoke --range 11-12 --verbose  # show detailed error messages

$ test/spec.sh smoke -r 11-12 -v  # short flags

In general, run test/spec.sh FOO to run the cases in spec/FOO.test.sh. Example:

$ test/spec.sh array       # run spec/array.test.sh

Run C++ tests

$ test/spec-cpp.sh run-file smoke  # one file
$ test/spec-cpp.sh osh-all

Also see our README.md: https://github.com/oilshell/oil/blob/master/spec/README.md

Note: test/spec-bin.sh setup is required to make all pass

See the comments at the top of that file for details.

You may be able to make small changes to Oil without this step. Generally, test/spec.sh smoke is a good indication that the shell is working.

But the whole test suite is very detailed and require specific versions of shells. So to make all spec tests pass, you have to download code and build it, as described at the top of test/spec-bin.sh.

Test-Driven Development

The idea behind the spec tests to figure out how OSH should behave (the spec) by taking an automated survey of the behavior of other shells. I follow a test-driven process like this:

  1. Write spec tests for a new feature.
  2. Make the spec tests pass on every shell except OSH. If shells differ in behavior, this may require annotations on the expected results.
    1. A given shell may not implement a feature. For example, bash and zsh both implement the dirs builtin, but mksh and dash don't.
    2. Shells may implement the same feature differently. For example, pushd in bash prints the stack to stdout, but pushd in zsh doesn't.
  3. Write code in OSH to make the tests pass.

After step 2, all columns should be green or yellow, except OSH. After step 3, the OSH column should be green or yellow as well.

Format of the Tests

The spec/foo.test.sh files are designed to be syntax-highlighted like normal shell scripts.

Each line is a "token". Lines with four hashes #### begin a test case. Lines with two hashes ## add metadata to the test case, e.g. assertions to make on status/stdout/stderr.

Basic Test:

#### test for echo
echo 1
## status: 0
## stdout: 1

Test with multiline assertion:

#### multiline test
echo 1
echo 2
## status: 0
## STDOUT:
1
2
## END

You can also add qualifiers to tests, to account for the different behavior of different shells. Example:

#### multiline test with qualifier
echo 1
echo 2
if test $SH = dash; then
  echo 3
fi
## status: 0
## STDOUT:
1
2
## OK dash STDOUT:
1
2
3
## END

You can also write assertions as JSON:

#### test for echo with tab
echo -e '1\t2'
## status: 0
## stdout-json: "1\t2\n"

$PWD and Creating Temporary Files

The current directory in the spec tests is the repo root. Make sure to cd $TMP first if you create any temporary files. (Cleaning them up at the end doesn't work because Ctrl-C could interrupt the tests.)

#### test with temporary file
cd $TMP
touch foo
test -f foo
## status: 0

Notes

  • Spec tests run in a semi-isolated environment. It could be more isolated (issue 42).
    • Almost all tests should pass if you run them on an Ubuntu machine with bash 4.4 (e.g. Ubuntu 18.04).
    • However, the tests tickle changes in minor versions of all shells, so to get everything to pass, you should follow the instructions at the top of test/spec-bin.sh. This builds shells from source.
    • Right now I run them on Ubuntu 16.04 with the above binaries, but they should run on any Linux distro.
    • OS X: TODO. This has been tried but people are not regularly using it.
  • It's OK to check in tests that don't pass on OSH yet. This helps because it specifies the behavior we want to implement. However, spec tests should not be submitted until they are green/yellow on OTHER shells. (They can be disabled if the feature isn't implemented at all in a shell.)
    • To prevent failing test runs, adjust --allowed-failures in test/spec.sh. For example, --allowed-failures 3 will make the sh_spec.py framework exit 0 if there are exactly 3 red failures in the OSH column. We only releaes OSH when all spec tests exit 0.
Add a custom footer
Add a custom sidebar
Clone this wiki locally