Release automation: run spec tests with just OSH on different platforms

[andychu]

For OS X, BSDs, etc. where we might not want to bother building with spec-bin.sh

And maybe ARM Linux would be a good one too. Although it would be nice to build binaries there.

[andychu]

This is is more significant now that we have a __GLIBC__ test in native/libc.c for extglob.

In other words, some things don't work on Alpine now, and it would be nice to see that in the tests.

[andychu]

TODO: What are dependencies of spec tests?

• Python 2
• bash
• coreutils (including nproc add portable nproc shim #415)
• gawk!

Anything else?

On the one hand, since they are confined to test/spec.sh and test/spec-runner.sh, we could make these more like POSIX scripts. I guess it depends what we do with Python 2, which is needed to run test/sh_spec.py. And gawk!

On the other hand, if we can use the same portability solution for all 4 deps, that would be nice.

Note test/spec-bin.sh builds binaries of other shells. We don't technically need it for this task. However we could use the same solution to build Python 2, bash, coreutils (which is admittedly onerous, but I guess BSDs are source-based anyway)

[andychu]

@jyn514 I realized the spec tests have a fairly significant gawk dependency. The match() function is a useful extension -- see test/spec-runner.sh

So ideally we could package up these 4 dependencies somehow. I don't think they are particularly weird or exotic.

I think putting gawk in $PATH is not too hard, as well as accounting for name differences with a tiny shell wrapper.

[andychu]

I think having something like:

$REPO/
  sh-env/
    darwin/
      nproc
      gawk
      rm
      xargs
    freebsd/
      nproc
      gawk
      rm
      xargs
    debian/
      # missing dependencies are assumed to be on system

would work. Some of these can be symlinks, some are tiny shell wrappers, and some are native binaries.

It's similar to the philosophy of avoiding #ifdefs. The Go/Plan 9 guys advocate that -- they avoid #ifdefs and instead create portable interfaces.

[andychu]

This is kind of the opposite of #42 -- because we want to run at least some tests in a DIFFERENT environment!

[andychu]

Updated dependencies:

For sh_spec.py, test/spec-runner.sh and dependencies like test/common.sh, and test/spec-alpine.sh, etc.

• Python 2 for sh_spec.py
  • this was in Python 3 at some point, but I dropped support for it, woudln't be too hard to port again. I think we mostly need b'' everywhere.
• bash or known working version of Oil (might have arrays, associative arrays, etc. so not just POSIX)
• gawk: probably could move this to Python pretty easily. For generating spec test HTML report, which is nice to upload.
• POSIX cp, rm, etc.

OPTIONAL:

• GNU findutils for xargs -P parallelization

---

tarball layout:

oil-0.8.pre2/
  _bin/
    oil   # oil-native, the thing being tested
    oil.ovm  # also can be tested
  test/
    spec-alpine.sh
  spec/
    ...
    var-op-other.test.sh
    ...
  

---

Problems expected

• assuming hardcoded paths like /lib exist
• assuming rm is in /bin/rm
• assuming that we have the Python repo, e.g. core/completion.py
  • (this could be in the tarball as TESTDATA then?)

[jyn514]

I think you'll also need python-dev since you include Python.h during build/dev.sh all. Not sure if spec tests run with the dev build or the tarball.

[andychu]

This is for testing the end-user binaries, so the user just needs to run ./configure && make etc.

No Python is needed to use Oil, either now or in the near future with oil-native! Developing Oil is a different story, but this issue is basically moving spec tests from "dev" --> "distro maintainer" / "end user"

[andychu]

TODO

• fold in test/alpine2.sh into test/alpine.sh
• Make it so you can copy an arbitrary tarball in, not just the _release/oil-*.tar.gz one
• set up non-root user so more tests pass
  • although the root user also exposes bugs!
• "diff" functionality vs Ubuntu spec results
  • look into failures. Some tests need to be modified. Categorize them?
• Make sure other people can successfully run it. Document it

Later:

• Run on OS X. This seems pretty feasible now
  • gawk is a remaining problem. Otherwise everything should be there: Python 2, bash, etc. and zip and scp for uploading

[andychu]

#70 is really a specific instance of this issue

[andychu]

Pinging this since it came up in the #oil-dev meeting

i.e. verifying a tarball/binary built with Nix