Coprocess Protocol V2

April 2021: This is OLD (2018). See Capers

(Back to Shell Autocompletion)

Related: Coprocess Language Support

Design Constraints

• Command line tools written in any langauge should be convertible to coprocesses.
• The patches required to do this must be small and localized.
  • For example, we shouldn't require modifying every print statement in the program and its dependencies!
  • There shouldn't be any special cases, e.g. for handling stdin, stderr, EOF, etc.

The initial Coprocess Protocol Proposal would require too much modification to existing tools. It was more like my 2012 projects fly and xmap. That only handled command line tools that behaved in a particular way.

Observations

• We do not control every print statement in the program.

  • Python interpreter prints stack traces to stderr.
  • Most VMs, e.g. Python or the JVM, print their own messages to stderr in rare cases, e.g. in out of memory conditions.
  • if you build with ASAN, this should work too.
  • It would be annoying to change every single print() statement in a Python program to print(msg, file=fcli_out).

• Just knowing sending stdin/stdout/stderr

Both of these should work, and they are different:

foo.py < in.txt >out.txt 2>err.txt

foo.py < in.txt >out-err.txt 2>&1

In the first case, the process is started with three files connected to its 3 descriptors.

In the second case, the process is started with two files connected to its 3 descriptors.

Configuration for FCLI Binaries

• concurrency policy
• env vars to pass -- all, none, or a specific list
• whether we should use descriptor passing or copying with named pipes (e.g. for pure Python 2 processes without recvmsg)
  • is there another protocol for JVM and node.js? i.e. things without dup2()? Or it sounds like you can do setOut(...)
• falling back to the batch client? Does the user need the option to use redo-ifchange in batch mode?
  • the real binary lives in /usr/bin, and wrapper lives somewhere else on $PATH? Like in ~/bin ?

What Clients Look Like

Clients for Shells Without FCLI Support

#!/bin/sh
exec fcli-invoke --fcli-socket /tmp/redoifchange -- redo-ifchange "$@"

Oil Client

We can save the exec() of fcli-invoke. Just build it in. Maybe that should just be a builtin so it can behave both ways?

TODO

• Signals? If you Ctrl-C in the client process, can it kill the whole coprocess?
• Unexpected exit by server
• Process Management
  • Do we need a lock file to prevent concurrent requests to the pipe?
    • Each shell instance could open its own?
  • What is the command to shut down a coprocess?
    • I think it should just be ~/fcli/ or something. or ~/.local/ ?
• Configuration for concurrency

Prototype

See https://github.com/oilshell/shell-protocols/tree/master/coprocess

Open Issues

• Can JVM call dup2() ? Can it recvmsg() a file descriptor?