HYSH

HYSH stands for Huang Ying’s SHell in Common Lisp.

Bash interactive shell and scripts are very important tools to use Linux. But I don’t like the syntax of bash, would rather to do that in Common Lisp. I read some document of SCSH and tried inferior-shell. Both are good and useful tool. I like Common Lisp and the capability to write pipeline filter in lisp (scheme) of SCSH. After tried to implement an iolib backend and add Common Lisp pipeline filter support to inferior-shell, I still feel that the command and redirection syntax of inferior-shell is just a direct translation of that of bash. Maybe I can do that in a more lisp style. It is the HYSH. The initial version was based on fork, later on I found that does not work well with thread, which I think is more popular programming model for Common Lisp, so I reimplemented it with thread and lfp-spawn of iolib.

Example

(defun dpkg-installed (&key (packagenames t))
  (out/lines
    (pipe (run dpkg -l)
          (if (eq packagenames t)
              (run cat)
              (run egrep (apply #'uiop:strcat
                                `("("
                                  ,@(iter (for (name . more) :on packagenames)
                                          (collect name)
                                          (when more (collect " | ")))
                                  ")")))))))

or write the filter in Common Lisp.

(defun dpkg-installed (&key (packagenames t))
  (out/lines
    (pipe (run dpkg -l)
	  (if (eq packagenames t)
	      (filter-line #'identity)
	      (filter-line (lambda (line)
			     (when (some (lambda (pkg)
					   (search pkg line))
					 packagenames)
			       line)))))))

The power of HYSH is that you can use Common Lisp to write pipeline filters and glue external programs and Common Lisp codes in various way.

Common conventions

There are mainly two categories of functions/macros in HYSH. Some functions/macros compute, some other functions/macros setup environment (IO redirecting, gluing with pipeline etc.) for computing.

Functions/macros to setup environment for some computing have one to several parameters to specify the one to several computing (for example, pipe). Most computing functions/macros return one to several value, the last return value indicates whether the computing is successful. Most functions/macros to setup environment will return the return values of one of the given computing functions/macros and may prepend some new values to the return values. For example, out/ss will prepend stdout output string to its computing’s return values.

Run external program

Unlike inferior-shell, HYSH provides only a very basic run function, you can specify just command line, because all IO redirection and glue between programs are done in Common Lisp functions/macros.

IO redirection

IO redirection in Unix process world means replace the original standard input, output, etc. file descriptors with file, pipe, etc. In HYSH, we defined IO redirection for Common Lisp. That means replace the original *standard-input*, *standard-output*, *error-output*, etc. streams with other streams of file, pipe, etc. So for an IO redirection, a Common Lisp stream special variable and a file descriptor number can be specified. After that is done, all Common Lisp function will reference the replaced streams for the special stream variables, and the specified file descriptors redirection will be setup for the external programs too.

Task

The task in HYSH is used to represent a Common Lisp thread or Unix process. There are parent/child relationship between tasks. And the child tasks will inherited all IO redirection, environment variables, and current directory, etc. from their parent tasks.

Glue between tasks

The most important glue is pipeline. I think this is the flagship of UNIX worlds. Now we can do that in Common Lisp. Any tasks can be connected with pipeline, regardless Common Lisp thread or Unix process.

Other glue mechanisms are provided too, including and, or, background, sequence.

External program error processing

Error processing in Common Lisp is done by conditions. A special continuable condition is defined for external program exiting with non-zero status, that is, failed in UNIX sense.

Arbitrary combination

The power of HYSH is that it provide a more flexible way to combine external programs, IO redirection, glue (pipeline, etc.), etc. with Common Lisp.

For example, to encapsulate some external filter program in Common Lisp with string as input and output. It can be accomplished with:

(in/s input-string (out/s (run filter arg1 arg2 ...)))

For given input-string and arguments, the form will return the result string and exit status of filter.

Comparison with INFERIOR-SHELL

For me, scripts in INFERIOR-SHELL are more like bash script in SEXP syntax, while scripts in HYSH are more like normal lisp program. For example, a script in INFERIOR-SHELL:

(run '(progn ls / (> output) (echo "abc" (>> output))))

vs. the script with similar functionality in HYSH:

(with-redirect-stdout-to-file "output"
  (run ls /)
  (format t "abc~%"))

Install

HYSH depends on the following ASDF systems, they can be installed via quicklisp.

• alexandria
• iterate
• split-sequence
• iolib
• bordeaux-threads

Download HYSH and the following systems and put them into a directory which can be found by ASDF (howto).

• hy-utils
• hy-stream