Unix Shells

The word shell itself was coined by Louis Pouzin after implementing RUNCOM (from which we get the fossil term/suffix rc) at MIT in the early-mid 1960s for its CTSS. He wrote a paper on how to implement it for Multics before returning to his native France. Glenda Schroeder subsequently implemented the Multics shell the predecessor of the Unix shell today.

There are a few Unix shells out there. I’ve used Bash since, it feels like, forever and so it’ll be Bash that I’ll be using for examples.

I’m by no means an expert in Bash, I think we all settle into our favoured idioms and every time I read the man page I came away with something new.

GNU’s Bash (first released in 1989), the “Bourne Again shell” is, of course, a nod to Stephen Bourne’s Bourne shell (first released in 1979) created as a free software alternative and picking up a few features from elsewhere. You used to get a Bourne shell when you logged in on a (proprietary!) Unix box. The Bourne shell was a replacement for Ken Thompson’s Thompson shell (introduced in the first version of Unix in 1971) which gave us our basic command orchestration syntax:

< in-file command1 | command2 > out-file

I can’t help but think that that original choice of the <, > and | symbols creating the visually pleasing < | > structure is a work of genius. Or a stroke of luck. Hard to say. Still, well done, Ken. His name rings ma bell, mind, has he done anything else?

You can read his 1976 paper on The Unix Command Language and its key notion of “I/O streaming and interconnection of utility program.”.

The Bourne shell subsequently added features like:

  • command substitution using backquotes

  • here documents

  • for loops

  • case statements

and, perhaps most intriguingly, the use of file descriptor 2 for error messages.

I can’t deny having had Bill Joy’s C shell (first released in 1978) as my default shell for a while as a youth – never forget Tom Christiansen’s 1995 Csh Programming Considered Harmful. Ken Greer’s tsch (first released in 1981) added Tenex-style file name completion to the C shell which Bash subsequently re-imagined.

I wrote a lot of code in David Korn’s KornShell (first released in 1983) when it was the most reliably-available shell across various development, test and production systems and through several substantial operating system upgrades.

I’ve barely touched either of Kenneth Almquist’s Ash (first released in 1989 again as a free software alternative to the Bourne shell) or its derivative, Herbert Xu’s Debian Almquist shell, Dash (first released in 1997) which appears in many embedded systems through Busybox and more recently as /bin/sh in Ubuntu and Debian systems.

I’ve not used Paul Falstad’s Zsh (first released in 1990) – although I see Mac OS X is encouraging me to switch, No, Mac OS X, no! – and so neither its popular “Oh My Zsh” collection of plugins and themes.

There’s a bit of a theme, there, though. Original proprietary Unix shells written in the late 70s or so and a suite of free software replacements written in the late 80s. Does that mean there’s been no new shell for thirty years?

No, of course not. What we haven’t had is any major distribution changing their default interactive shell or default shell alternatives.

Axel Liljencrantz’s friendly interactive shell, fish (first released in 2005) is focused on usability and interactive use.

Andy Chu, chubot‘s Oil shell (first released in 2017) is trying to be a better POSIX shell for programmers.

Laurence Morgan’s Murex shell (official release pending) is designed for DevOps productivity.

Ilya Sher’s Next Generation Shell NGS (from 2013) where he’s looking at the same annoyances as I do (poor data structures in the shell and a lack of system administration focus in general purpose languages).

Alessandro Nadalin’s ABS programming language (from 2018) “bringing back the joy of shell scripting”.

There are a slew of shells featuring the idea of passing structured data through pipelines – something made most famous by Microsoft’s PowerShell.

  • Andrew ChambersJanetsh is an interactive shell and scripting tool based on the Janet programming language (which has a Lisp-ish notation).

  • Qi Xiao’s elvish (first released in 2017) is an interactive shell and programming language.

  • Matt Russell’s Mash (2016 - 2017?) is another object-passing shell.

  • Jonathan Turner’s Nushell (first released in 2019) is another structured data pipelining shell.

  • Jack Orenstein’s Marcel (the name appears to be from the video short, Marcel the Shell with Shoes On, enjoy!) (from 2020?)

Stepping away from regular shells, Olin ShiversScsh (first released in 1994 although development may have stalled in 2006) is a shell embedded in Scheme together with a syscall library for systems programming.

Following the idea of a programming language with shell-like syntax, there’s Anthony Scopatz’s Xonsh (from 2015) which implements a shell in Python and/or Python in the shell.

There are even papers being written about The Once and Future Shell.

The Interactive Shell

The shell can be described as two quite different things. On the one hand there is a (textual) user interface where users input shell commands at a prompt, a form of REPL. There’s a huge opportunity here for customising the look and feel of an interactive shell from the prompt through command editing, command and argument completion and onto history replay.

On the other hand, when a shell is not interactive, it’ll be reading commands from a script and isn’t concerned about prompts, history or completion.

I’ll state quite clearly that we’re not here for the interactive/user interface functionality of the shell. We’re looking at shell scripting and if we can make better shell programs. Bash, Zsh and fish and any number of others have more than we need for “↑ RETURN” or whatever fancy interaction with the shell you care to name.

This is partly because we’re unlikely to do a better job but mostly because it’s hard. It’s really hard. I can’t find a reference for it any more but one of the man pages for terminal-related functionality (termio?) used to note that figuring out the terminal is probably the hardest thing a user had to do which was unfortunate because it was one of the first things a user had to do.

The guys who are making the use interaction more pleasant are still discovering bugs in decades-old terminal drivers behaving badly requiring all sorts of artistic solutions – look at Zsh and Fish’s simple but clever trick for highlighting missing linefeeds and the follow-up commentary in https://news.ycombinator.com/item?id=23520240. The problem revolves around the nicety of handling a program that doesn’t complete its output with a newline:

% echo hello
hello
% echo -n hello
hello%

Notice that my prompt, % , appears at the end of the second command’s output, in column 5, in this case, not on a clear(ed) line like the others.

You, the shell, don’t know what the program printed so, broadly, the trick is to always print a “missing newline” glyph (%, ¶, ⏎ or something) then some terminal subtleties: print $COLUMN-1 spaces; carriage return and then print out the prompt (and a clear-to-end-of-line to flush the remaining spaces).

  • If the program emitted a trailing newline then the above will have printed the glyph in column 0 and spaces up to the end of line and then the carriage return will have brought you back on top of the glyph. The shell’s prompt will over-write the glyph and everyone is happy.

  • If the program didn’t print a trailing newline then the output will have been followed by the glyph and enough spaces to take you onto the next line of the terminal. The carriage return takes you back to column 0 of this next line and the shell prints its prompt leaving the impression of the glyph at the end of the text:

% echo hello
hello
% echo -n hello
hello⏎
%

That is the least of it, though, as the real problems come when asserting what any given terminal will do when you print a character (here, one of those spaces) in the last column of the line. The cursor goes in “the next” column, right? Which, given that we’ve just printed something in the last column means we wrap round onto the next line (with an implied newline). Er, nope. You’ve obviously missed out on the joys of clanking away on an electro-mechanical teletype or one of its VDU successors. There are terminal capabilities that describe various end-of-line behaviours and then there’s bugs in the terminal’s behaviour.

We don’t need to go there so let’s not. That’s not to say we’ll do nothing with the terminal – we are required for job control to mess around a little bit with terminals – but there’s enough to do as it is. Let’s not get distracted.

Meh! Except we might want to get distracted, eventually. In an ideal world I suspect we’ll have incorporated a table-driven reader by then which would (probably) make life a lot easier. In the meanwhile, we haven’t got one of them and dealing with the terminal sounds like it’ll be a pain.

Oh but we could really do with something, mind you, as I keep typing Ctrl-P and feeling like I’ve fallen into vi-mode. It is… challenging …to have no editing options at the prompt. We can’t readily incorporate (our men) Chet Ramey (after Brian Fox)’s Readline library thanks to our slightly pedagogical stance of not incorporating things we can’t explain. No black boxes.

(You can run rlwrap, of course.)

Last built at 2025-01-10T07:11:03Z+0000 from 463152b (dev)