Chapter F.0: The Unix Survival Kit§

Authors: Shrey Patel and Jay Patel, Coconut Labs
Book: Think Like an OS (see book/00-INDEX.md)

This chapter gets you fluent enough in a Unix terminal that nothing later in the book stalls on mechanics. By the end you will know what the shell actually is (a small programming language, not a magic box), where things live on a Linux disk and why, how permissions and processes look from the user side, how to reach a remote machine over ssh, and what really happens when you type ./configure && make && make install. The lab has you build a real C project from a source tarball, poke at a live process through /proc, and answer a question about a log file using nothing but pipes. If you have never lived in a terminal, this is your on-ramp. If you have, the Bridge notes at the end tell you which 40% you should still read.

The problem§

Every later chapter in this book assumes you can drive a Unix system from a text prompt. Not expertly. Just without fear. The chapters on syscalls assume you can run a program and check its exit code. The chapters on memory assume you can read a file under /proc and know it is not a real file. The kernel-build chapters assume make is a familiar verb. The debugging chapters assume kill -9 means something specific to you, and that you know why it is the last resort rather than the first.

Two kinds of reader end up here, and the chapter is load-bearing for both. If you are heading for kernel work, the terminal is the only interface the kernel has to you: you build it with make, boot it at a serial console that is nothing but a prompt, and read every complaint it has as text. If you are heading for low-latency or trading systems, the box that runs your code is a Linux machine you will only ever reach over ssh, and when it misbehaves the evidence is in /proc, in an exit code, and in which signal killed what. The commands are the same commands.

If your daily environment is macOS or Windows with GUI tools, none of this is obvious, and that is not a gap in your intelligence. It is a gap in exposure. GUIs deliberately hide the layer this book is about. The terminal is where the operating system stops hiding. Processes, files, permissions, signals, devices: every concept in this book has a direct, touchable representation in the shell. Learning the shell is not a chore before the real content; it is your first contact with the real content.

Unfolded§

The shell is a programming environment§

When you open Terminal on macOS or start a Linux container, a program called a shell starts, prints a prompt (usually something ending in $ or %), and waits. The shell is an ordinary program whose job is to read a line of text from you, interpret it, run other programs on your behalf, and show you their output. Bash and zsh are the two shells you will meet most; everything in this chapter works in both.

A command line has a simple anatomy:

ls -l /tmp
│  │  └── argument: a path the program is given
│  └───── option (also an argument): "-l" asks for long format
└──────── the command: a program named "ls"

The shell splits the line on spaces, treats the first word as the program to run, and passes the rest to that program as its arguments. The program decides what the arguments mean; the shell just delivers them.

How does the shell find a program named ls? Through an environment variable called PATH. Environment variables are named strings every process carries around; a child process inherits a copy of its parent's environment. PATH holds a colon-separated list of directories, and the shell searches them left to right, running the first match:

echo $PATH

Expected: something like /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin. The details vary per machine.

echo prints its arguments; $PATH tells the shell to substitute the variable's value. You can see any variable with echo $NAME, list them all with env, and create your own with export NAME=value (the export makes it visible to child processes, not just the shell itself). If you type a command and get command not found, either the program is not installed or it lives in a directory not on your PATH.

Composition is what the shell is for. Every program gets three standard streams: standard input (stdin, stream 0), standard output (stdout, stream 1), and standard error (stderr, stream 2). By default all three connect to your terminal, but the shell can rewire them before the program starts:

That last row, the pipe, is the whole Unix philosophy in one character. Small programs, each doing one job, chained into pipelines:

┌────────┐ stdout→stdin ┌────────┐ stdout→stdin ┌────────┐
│  grep  │─────────────▶│  sort  │─────────────▶│   wc   │
│ filter │              │ order  │              │ count  │
└────────┘              └────────┘              └────────┘

Finally, every program reports how it went. When a process exits it returns a small integer, its exit code: 0 means success, anything else means failure. The shell stores the last command's code in $?. Three conventions from the bash manual are worth memorizing: 127 means "command not found," 126 means "found but not executable," and if a process is killed by signal number N, the shell reports 128+N. Exit codes power shell logic: a && b runs b only if a succeeded; a || b runs b only if a failed. That is all ./configure && make && make install is: three commands where each runs only if the previous one worked.

The filesystem, unfolded§

Unix has one directory tree, rooted at /. There are no drive letters; disks, partitions, and even network shares are mounted (grafted) onto directories inside the one tree. The layout is standardized by the Filesystem Hierarchy Standard (FHS, version 3.0, published 2015), so the same instincts work on any Linux box:

/
├── bin/    essential user command binaries (ls, cp, sh ...)
├── etc/    host-specific system configuration (text files)
├── usr/    the "second hierarchy": shareable, read-only data
│   ├── bin/     most installed programs
│   └── local/   software the local admin built or installed by hand
├── var/    variable data: logs (/var/log), spools, caches
├── home/   users' home directories (/home/alice)
├── tmp/    temporary files, often wiped at boot
├── dev/    device files - hardware exposed as files
├── proc/   virtual: kernel and process information
└── sys/    virtual: kernel objects, devices, drivers

A few of these deserve a longer look.

/etc is where a Unix system keeps its configuration, as plain text files you can read and edit. /var holds data that changes as the system runs; /var/log is where you go when something misbehaves. /usr holds shareable, read-only data: the operating system's installed software. /usr/local is reserved for things you build yourself, which is exactly where make install will put our lab project.

/dev embodies a founding Unix idea: devices are files. /dev/sda is a whole disk; reading it reads raw sectors. /dev/null is a black hole that discards writes (you will often see 2> /dev/null to silence errors). /dev/urandom produces random bytes. The point is uniformity: the same open/read/write operations that work on documents work on hardware.

/proc and /sys go one step further: they are windows into the running kernel. Neither exists on disk. The proc filesystem is, in the words of its manual page, "a pseudo-filesystem which provides an interface to kernel data structures." /proc/cpuinfo describes your CPUs; /proc/meminfo your memory. Every process gets a directory /proc/<pid> containing its command line, environment, open files, and memory map. That is the kernel's live bookkeeping, formatted as text, readable with cat. /sys (sysfs) is the tidier sibling: a RAM-based filesystem exporting kernel objects and their attributes, with a strong convention of one value per file. When later chapters say "the kernel tracks X per process," /proc is where you can go and watch it do so.

If it feels wrong to cat a file that has no bytes on any disk, that reaction is correct rather than confused. The kernel assembles the contents while you read them. macOS has no /proc at all, which is one reason the lab below runs in a Linux container.

Users, groups, permissions, sudo§

Unix was multi-user from the start, and the permission model shows it. Every user has a numeric ID (UID); UID 0 is root, the superuser, who bypasses permission checks. Users belong to groups. Every file has one owning user and one owning group, plus nine permission bits: read, write, and execute, each granted separately to the owner, the group, and everyone else.

ls -l shows all of this in one dense string:

-rwxr-xr--  1 alice staff  8192 Aug 14 10:00 deploy.sh
│└┬┘└┬┘└┬┘    │     │
│ │  │  └ other: r--   (read only)
│ │  └── group: r-x   (read, execute)
│ └───── owner: rwx   (read, write, execute)
└─────── type: "-" file, "d" directory, "l" symlink

For a regular file, r means you can read it, w means you can modify it, x means you can run it as a program. For a directory the meanings shift: r lets you list the names inside, w lets you create and delete entries, and x lets you enter it and reach things through it.

Each rwx triplet maps to an octal digit: r=4, w=2, x=1, summed. So rwxr-xr-- is 754, and the two you will see constantly are 755 (rwxr-xr-x, a typical program or directory) and 644 (rw-r--r--, a typical document). chmod 755 file sets the bits; chown alice:staff file changes ownership.

When you need root's power briefly (installing into /usr/local, editing /etc), you use sudo cmd, which runs one command as root after checking that your user is authorized. The discipline to build now: run as yourself by default, escalate per command, and be suspicious of any instruction that begins with sudo you don't understand.

Processes from the user side§

A process is a running instance of a program, with its own memory, its own ID (PID), and a parent that spawned it. Two viewing tools:

ps aux        # snapshot of every process: user, PID, CPU%, MEM%, command
top           # live-updating view; press q to quit

htop is a friendlier top (colors, scrolling, tree view) but usually needs installing.

You talk to processes with signals, tiny numbered notifications the kernel delivers. A process can install a handler for most signals, ignore them, or accept the default action. The ones to know (numbers as on x86/ARM; a few differ on other architectures):

The kill command sends signals (its name oversells it). kill 1234 sends SIGTERM, the default when no signal is specified, giving process 1234 a chance to clean up and exit. kill -9 1234 (or kill -KILL 1234) is the uncatchable hammer. Reach for TERM first, KILL only when TERM is ignored, because a KILLed process gets zero opportunity to flush data or release resources. kill -l lists every signal name.

The shell also does lightweight process management called job control. Append & to run a command in the background: long_build & returns your prompt immediately, and $! holds the new PID. jobs lists background jobs, fg brings one to the foreground, Ctrl-Z stops (freezes) the foreground job so you can bg it to resume in the background. One catch: when your terminal closes, the kernel sends SIGHUP to your shell (the session leader) and the foreground job. An interactive shell like bash then resends SIGHUP to all its jobs before exiting, so background jobs die too, killed by default. nohup cmd & starts a command immune to hangups, appending its output to nohup.out (or $HOME/nohup.out if the current directory is unwritable). That is the classic way to leave work running after you log out.

ssh: your hands on a remote machine§

ssh gives you a shell on another machine over an encrypted connection. ssh alice@server.example.com logs in as alice and drops you at a prompt on the remote machine; everything in this chapter works identically there.

Skip passwords and use keys. ssh-keygen generates a keypair. Since OpenSSH 9.5 the default type is Ed25519. You get a private key ~/.ssh/id_ed25519, which never leaves your machine and never gets shared, and a public key id_ed25519.pub, which is safe to publish. Append the public key to ~/.ssh/authorized_keys on the server (ssh-copy-id alice@server automates this) and subsequent logins authenticate with the key.

~/.ssh/config saves your fingers:

Host build
    HostName server.example.com
    User alice
    IdentityFile ~/.ssh/id_ed25519

Now ssh build does the whole dance. To copy files, scp local.txt build:/tmp/ pushes and scp build:/var/log/app.log . pulls. Same syntax as cp, with host: prefixes.

Building software from source§

The classic incantation is three commands glued by &&, so each runs only if the last succeeded:

./configure && make && make install

./configure is a portable shell script shipped in the tarball (generated by a tool called Autoconf from a recipe the developers wrote). It interrogates your system: which compiler exists, which headers and libraries are present, how big types are. Then it writes out a Makefile tailored to what it found, plus typically a config.h of feature flags and a config.log you can read when it fails. Its most useful flag is --prefix=DIR, which sets where installation will land; the GNU default prefix is /usr/local.

make reads the Makefile, which declares targets, their dependencies, and the commands to build each target. make builds the dependency graph and runs only what is out of date. Touch one .c file and only that object recompiles. This incremental property is why make (or a descendant) sits under virtually every build system you have used.

make install copies the built artifacts into the prefix: binaries to /usr/local/bin, manuals to /usr/local/share/man, and so on. Because /usr/local is root-owned on most systems, this is the step that needs sudo. The previous two should run as you.

Newer projects use newer generators with the same shape: configure once, then build.

CMake and Meson both generate files for a fast backend (commonly Ninja) instead of driving compilation themselves. The mental model is identical: probe the system, generate build rules, build incrementally, copy into a prefix.

The text-tool survival kit§

Five tools cover most of what this book's labs need:

tar ("tape archive") bundles a directory tree into one file; the z runs it through gzip compression. Source code is distributed as "tarballs" (compressed tar archives), which is why tar xzf is the first command of the lab. sed goes deep; the single substitute command above is 90% of daily use.

The real thing in Linux§

Everything above has a concrete address in the Linux 6.12-era source tree or its standard toolchain:

• The /proc/<pid> directories are implemented in fs/proc/base.c in the kernel tree; the authoritative catalog of every file under /proc is the proc(5) manual page. Try man 5 proc on any Linux box. Several /proc files (like cmdline) separate fields with NUL bytes, which is why the lab pipes them through tr '\0' ' '.
• /sys is sysfs, documented in the kernel's own tree at Documentation/filesystems/sysfs.rst: a RAM-based filesystem that exports kernel objects ("kobjects") and their attributes as directories and one-value-per-file text files.
• Signal delivery lives in kernel/signal.c. The userspace contract is signal(7): numbers, default actions, which signals are uncatchable.
• PATH search is not a kernel feature. The kernel's execve syscall takes an exact path; the search over $PATH is done in userspace by C-library wrappers like execvp/execlp (see exec(3)), or by your shell, which is just another userspace program doing the same walk.
• The everyday commands ls, cp, cat, tr, and nohup come from GNU coreutils; grep, sed, tar, make, and Bash are their own GNU projects. On macOS you get BSD-lineage versions with slightly different flags, a divergence you will notice the first time sed -i behaves differently.

Coconut tie-in§

Every Coconut OS artifact you will meet later in this book is operated exactly this way: kernel builds are make runs, boot tests happen at a serial console that is nothing but a shell prompt, and the agent_* syscall experiments are driven from a terminal inside QEMU. Coconut's audit trail (per 04-HLD) is JSON-lines, a format chosen precisely so the grep-and-pipe skills from this chapter interrogate it directly. And the /proc-style instinct of "the kernel will show you its bookkeeping as files" is the mental model behind inspecting Coconut's agent and capability state.

Lab§

Environment: a Docker Ubuntu container, because the /proc part requires Linux and macOS has no /proc. Everything below is meant to be typed, not read. Install Docker Desktop, then from macOS Terminal:

docker run -it --rm ubuntu:24.04 bash

You are now root inside a disposable Ubuntu machine; exiting deletes it. Install the tools (about 2 minutes):

apt-get update && apt-get install -y build-essential wget ca-certificates less procps

Part 1: exit codes (3 min). Predict what each echo $? prints, then run:

true;  echo $?
false; echo $?
nosuchcommand; echo $?

Expected: 0, then 1, then an error message followed by 127 (the shell's "command not found" code).

Part 2: build a real project from a tarball (10 min). Predict: after make install with no --prefix, which directory will the hello binary land in?

cd /tmp
wget https://ftp.gnu.org/gnu/hello/hello-2.12.3.tar.gz
tar xzf hello-2.12.3.tar.gz
cd hello-2.12.3
./configure
make
./hello

Expected: ./configure scrolls system checks ("checking for gcc... gcc") and ends by writing Makefiles; make scrolls compile lines; ./hello prints a hello-world greeting.

make install
which hello
hello

Expected: which prints /usr/local/bin/hello, the GNU default prefix from the chapter. (No sudo needed only because you are already root in the container.) Skim less config.log to see what configure actually probed; q quits.

Part 3: a process under /proc (8 min). Start a background process and inspect the kernel's view of it. Predict: what will State say for a process that is asleep in sleep?

sleep 300 &
echo $!
tr '\0' ' ' < /proc/$!/cmdline; echo
grep State /proc/$!/status
ls -l /proc/$!/fd
readlink /proc/$!/exe

Expected: the PID; then sleep 300; then State: S (sleeping); then three fd symlinks (0, 1, 2) pointing at your terminal device; then the path of the sleep binary. Now kill it and predict the exit code before running the last line:

kill $!
wait $!; echo $?

Expected: Terminated, and 143. That is 128 + 15, death by SIGTERM, exactly the convention from the exit-codes section.

Part 4: answer a question with pipes (7 min). Ubuntu logged every package your apt-get run installed to /var/log/dpkg.log. Question: how many packages were installed, and which three were installed last? Predict rough answers, then:

grep " install " /var/log/dpkg.log | wc -l
grep " install " /var/log/dpkg.log | awk '{print $4}' | tail -3

Expected: a count of several dozen (build-essential pulls in a lot), then three package names. The exact list varies with the image version. Read the pipeline aloud: filter the log to install events, then count; filter, project column 4, take the last three. Filter, transform, reduce is the composition model, and you will use it on kernel logs for the rest of the book.

Four parts in, you can start a process, find it in the kernel's own bookkeeping, read what state it is in, kill it, and explain the number that comes back. That loop is how every stall in this book gets diagnosed, and it is the same loop whether the process is a kernel build or a market-data handler on a box you cannot see.

Cleanup: exit. The container and everything in it vanishes.

Bridge notes§

If you have compiler-design and microprocessor coursework, roughly 60% of this chapter is muscle memory already. Calibrate like this:

• Skim: shell anatomy, redirection tables, permission bits, ps/top, ssh basics. You know these.
• Do not skip /proc and /sys. Coursework rarely covers the fact that the kernel exports live per-process state as a pseudo-filesystem, or that /sys is the one-value-per-file interface to kobjects. Later chapters use both as observation instruments, and the lab's wait $!; echo $? → 143 chain is a two-line demo of kernel/shell interface conventions worth internalizing.
• Do not skip signals by name and number. You likely know signals as a concept; the working vocabulary (TERM vs KILL semantics, 128+N exit encoding, which two signals are uncatchable) is what the debugging chapters assume.
• Do not skip build-from-source. Knowing how a compiler works is different from knowing what ./configure generates, why the prefix defaults to /usr/local, and how CMake/Meson map onto the same three phases. The kernel-build chapters lean on this directly.
• Worth 60 seconds: the note that PATH resolution is userspace (execvp), not kernel (execve). That boundary is one students of both compilers and architecture often misplace.

Sources§

• https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html: FHS 3.0 (2015-03-19): purposes of /bin, /etc, /home, /dev, /proc, /tmp, /opt.
• https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04.html: /usr as shareable read-only secondary hierarchy; /usr/local as the local hierarchy.
• https://man7.org/linux/man-pages/man7/signal.7.html: signal numbers on x86/ARM; SIGKILL/SIGSTOP uncatchable; default actions.
• https://man7.org/linux/man-pages/man5/proc.5.html: proc as pseudo-filesystem interface to kernel data structures; NUL-separated fields; per-PID entries.
• https://www.kernel.org/doc/html/latest/filesystems/sysfs.html: sysfs as RAM-based export of kobjects/attributes; one-value-per-file convention.
• https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html: exit codes 126, 127, and 128+N for fatal signals.
• https://man7.org/linux/man-pages/man1/kill.1.html: TERM is the default signal; kill -l lists names.
• https://man7.org/linux/man-pages/man1/nohup.1.html: hangup immunity; nohup.out / $HOME/nohup.out fallback.
• https://www.gnu.org/prep/standards/html_node/Directory-Variables.html: GNU default prefix is /usr/local.
• https://www.openssh.org/txt/release-9.5: ssh-keygen generates Ed25519 keys by default since 9.5.
• https://mesonbuild.com/Quick-guide.html: meson setup / meson compile workflow; Ninja requirement.
• https://ftp.gnu.org/gnu/hello/: hello-2.12.3.tar.gz is the current GNU hello release used in the lab.
• https://man7.org/linux/man-pages/man3/exec.3.html: execvp/execlp search PATH in userspace, layered on execve(2).
• https://elixir.bootlin.com/linux/v6.12/source/fs/proc/base.c: /proc/PID implementation present in the v6.12 tree.
• https://elixir.bootlin.com/linux/v6.12/source/kernel/signal.c: signal delivery code present in the v6.12 tree.