Chapter 8: Processes & Threads§

Authors: Shrey Patel and Jay Patel, Coconut Labs
Book: Think Like an OS. See book/00-INDEX.md

After this chapter you can explain what a process actually is, and the answer is not "a running program" but a bundle of ownership the kernel tracks in one struct. You will prove it at a terminal: read task_struct without flinching, predict what fork() copies and what it lazily shares, manufacture and reap a zombie, and watch copy-on-write happen live in /proc/PID/smaps. You will also see where Coconut OS bolts a second ownership unit, the agent, onto this fifty-year-old machinery.

Two payoffs, depending on where you are headed. For kernel work, task_struct is the object almost every subsystem in the tree eventually touches, so learning to read it by concern instead of top to bottom is a skill you will use in every chapter after this one. For latency-sensitive work, this is the chapter where "process" and "thread" stop being interchangeable words: the kernel schedules tasks, the thing you name when you pin or measure one thread is its TID and not its PID, and the cost of fork() turns out to scale with how much memory you have mapped rather than with how much you use. Those are not trivia. They decide what a measurement means.

The problem [fundamental]§

A CPU executes one instruction stream per core and has no opinions about whose instructions they are. Left alone, the machine is a microcontroller: one program owns all the memory, all the devices, all the time. Fine for a thermostat; it collapses the moment you want two programs, because every hard question lands at once. If program A scribbles over program B's memory, who notices? If a program opens 10,000 files and crashes, who closes them? If it spins in a loop, who takes the CPU back? If it spawns helpers and dies, who cleans up the helpers?

Every one of those questions is really one question: who owns what, and who gets the blame? An operating system needs a unit of accounting: a boundary it can draw around a running program so that memory, open files, CPU time, permissions, and children all belong to something, and so that when that something dies, the kernel can walk the boundary and reclaim everything inside it. No leaked memory, no orphaned file handles, no unaccounted CPU.

That unit is the process. Threads, fork(), zombies, signals: all of it falls out of taking "unit of ownership" seriously. This chapter unfolds each piece.

Unfolded [fundamental]§

The process: a bundle with a name [fundamental]§

Forget "a program in execution" for a moment. A program is a file on disk, passive bytes. A process is the live bundle the kernel builds when it agrees to run those bytes:

┌─ Process (PID 1234) ────────────────────────────────┐
│                                                     │
│  Address space ──── the process's private memory    │
│                     map: code, heap, stacks, libs   │
│  Threads ────────── 1..N instruction streams, each  │
│                     with its own registers + stack  │
│  Credentials ────── who this is: user id, group     │
│                     ids, capabilities               │
│  File descriptors ─ table of open files, sockets,   │
│                     pipes: small integers 0,1,2...  │
│  Signal state ───── handler table + pending signals │
│  Limits ─────────── caps on memory, fds, CPU time   │
│                                                     │
└─────────────────────────────────────────────────────┘

Each piece exists to answer an ownership question. The address space is the process's private view of memory, virtual addresses that the MMU translates to physical pages (chapter 10's subject), so process A literally cannot name process B's memory. The credentials answer "what is this process allowed to do," and every permission check in the kernel reads them. The file descriptor table maps small integers to open files and sockets, so the kernel knows exactly what to close at exit. Limits cap consumption, and the threads are the part that actually executes.

The payoff is death. When a process exits, cleanly or by segfault, the kernel walks this bundle and releases everything: unmaps the memory, closes every descriptor, cancels timers, reparents the children. A process cannot leak resources past its own lifetime, because the bundle is the ledger.

Threads: tasks that share the memory bag [fundamental]§

Here is the part that surprises people coming from textbooks: Linux has no separate "thread" object in the kernel. The kernel schedules tasks. A task is one schedulable instruction stream: one set of saved registers, one stack, one entry in the scheduler's queues. A "process with four threads" is, to the kernel, four tasks that happen to share the same address space, the same file descriptor table, and the same signal handler table. A "single-threaded process" is one task that shares with nobody.

The sharing is explicit, flag by flag, at creation time, and we will meet the flags in a moment. The one that defines threadness is CLONE_VM: both tasks use the same memory map, so a write by one is instantly visible to the other. That is the whole reason threads are both convenient (pass a pointer, no copying) and dangerous (data races, chapter 12).

Naming gets subtle. Every task has its own kernel-level id (its pid field). Tasks created with CLONE_THREAD join the same thread group, named by its leader's id, the thread group id or TGID. What userspace calls "the PID" is the TGID: that is what getpid() returns, what ps shows, and what you kill. The per-task id is exposed as the thread id via gettid(). This is the distinction that decides what a per-thread measurement is attached to, so it is worth getting straight now rather than at 2am. A vocabulary table, since three names for two concepts is genuinely confusing:

You sayKernel objectId you see
processthread group of tasksPID (= leader's TGID)
threadone task in the groupTID (gettid())
single-threaded processgroup of exactly one taskPID = TID

fork: duplicate now, copy later [fundamental]§

Unix creates processes by duplication: fork() clones the calling process into a nearly identical child. Same code, same heap contents, same open files. It is the only syscall that returns twice: once in the parent (returning the child's PID) and once in the child (returning 0), which is how each copy learns which one it is:

c
pid_t pid = fork();
if (pid == 0) { /* I am the child  */ }
else          { /* I am the parent; child is `pid` */ }

Duplicating a multi-gigabyte address space sounds ruinously expensive. It would be, if the kernel actually copied the memory. Historically it did, which is why vfork() exists (below). Modern fork() cheats with copy-on-write (COW). Recall from the memory chapters that a process's memory map is described by page tables: per-page entries saying "virtual page X lives at physical page Y, writable." At fork, the kernel copies only the page tables, points both processes' entries at the same physical pages, and marks those entries read-only in both:

   Parent page table          Child page table
   ┌───────────────┐          ┌───────────────┐
   │ page 7 → P, ro│─────┐  ┌─│ page 7 → P, ro│
   └───────────────┘     ▼  ▼ └───────────────┘
                    ┌──────────┐
                    │ phys page│   one physical copy,
                    │    P     │   shared, read-only
                    └──────────┘
        ...until someone writes to page 7:
                    ┌──────────┐   ┌──────────┐
       parent ─────▶│    P     │   │    P'    │◀───── child
                    └──────────┘   └──────────┘
                        (writer gets a private copy; the
                         other keeps the original)

When either side writes a shared page, the CPU faults (the page is read-only), the kernel catches the fault, copies that one page, updates the writer's entry to point at the private writable copy, and resumes. Pages nobody writes are never copied at all. The fork(2) man page states it plainly: the only penalty is "the time and memory required to duplicate the parent's page tables, and to create a unique task structure for the child." You will watch this happen in the lab.

Read that penalty clause carefully, because it is the part people get wrong. The cost of fork() tracks the size of the page tables, which tracks how much address space the parent has mapped, not how much it is actively using. A process holding a large mapping pays for a fork it never writes a byte into. That is why the surprise usually shows up as a latency spike in a big long-lived process that forks occasionally, and never in the small one you tested with.

Not everything carries over. The child gets copies of the descriptor table and credentials, but starts with an empty set of pending signals, does not inherit the parent's memory locks or timers, and its resource-usage counters reset to zero. The ledger starts fresh.

vfork() is the fossil that shows why COW matters: it shares memory outright, with no copy at all, and suspends the parent until the child calls execve() or exits. It was a BSD-era optimization for fork-then-immediately-exec back when fork really copied everything; with COW it is rarely worth the sharp edges.

One engine underneath: clone and the sharing knobs [working]§

fork(), vfork(), and pthread_create() are not three mechanisms. They are three settings of one mechanism: clone(). Clone creates a new task and takes a bitmask of flags saying which parts of the bundle the new task shares with its creator versus gets copies of. The modern, extensible variant is clone3() (added in Linux 5.3), which takes a versioned struct instead of an ever-growing argument list.

The five flags worth memorizing:

FlagIf set, parent and child share...If clear, child gets...
CLONE_VMthe address space, so writes by one are visible to the othera COW copy of the memory map
CLONE_FILESone file descriptor table, so an fd opened by either is valid in botha copy of the fd table
CLONE_FSfilesystem info: root, current working directory, umaskcopies of those
CLONE_SIGHANDthe table of signal handlersa copy of the handler table
CLONE_THREADthread-group membership: child joins the caller's group, same PIDchild is a new process with its own PID

The flags compose, but not freely: since Linux 2.5.35 CLONE_THREAD requires CLONE_SIGHAND, and since 2.6.0 CLONE_SIGHAND requires CLONE_VM. That is the kernel encoding a truth: threads that share a PID must share signal handling, and shared signal handlers only make sense in shared memory.

Now the three "different" calls collapse into a table:

CallEffectively
fork()clone with no sharing flags: copy everything (lazily, via COW)
vfork()clone with `CLONE_VFORK \CLONE_VM`: share memory, suspend parent
pthread_create()clone with `CLONE_VM \CLONE_FILES \CLONE_FS \CLONE_SIGHAND \CLONE_THREAD` (and more): share everything, new stack

"Process vs thread" is not a kernel distinction. It is two corners of a configuration space.

execve: same bundle, new program [fundamental]§

fork() makes a new process running the old program. To run a new program, the child calls execve(path, argv, envp): the kernel throws away the current address space, loads the executable at path, builds a fresh stack with the arguments, and jumps to its entry point. Crucially, it is still the same process: same PID, same credentials (setuid aside), and open file descriptors survive by default.

That last property is why Unix splits creation into fork-then-exec rather than one "spawn" call: between the two, the child can rearrange its own bundle. A shell implementing ls > out.txt forks; the child opens out.txt, installs it as descriptor 1 (stdout), then execs ls, which writes to stdout never knowing it was redirected. The mechanics of loading (ELF headers, interpreters, the initial stack layout) are chapter 24's subject.

Lifecycle: states, zombies, orphans, and a race fixed 40 years late [fundamental]§

Between creation and reaping, a task is always in one of a few states, the letters you see in the STAT column of ps:

CodeStateMeaning
Rrunning/runnableexecuting, or on the run queue waiting for a CPU
Sinterruptible sleepwaiting for an event (network data, a timer); signals can wake it
Duninterruptible sleepusually mid-I/O; will not respond even to kill -9 until the wait ends
Zzombieterminated, but not yet reaped by its parent
Tstoppedpaused by a job-control signal (Ctrl-Z); t = stopped by a debugger
Iidleidle kernel thread (kernel-internal housekeeping tasks)
                fork/clone
                    │
                    ▼
        ┌────────► R ◄────────┐
        │      (runnable)     │ event arrives /
 signal │           │ waits   │ I/O completes
 (stop/ │           ▼         │
 cont)  ▼      S ── or ── D ──┘
        T      (sleeping)
                    │ exit()
                    ▼
                    Z ──── parent wait()s ────▶ gone
                (zombie)

The strange one is Z. When a process exits, the kernel reclaims its memory and descriptors immediately, but it does not delete the process record. It keeps a minimal stub: the PID, the exit status, resource-usage totals. Why? Because the exit status belongs to the parent, and the parent may not have asked for it yet. The parent collects it with wait() or waitpid(pid, &status, ...), which is what "reaping" means, and only then does the kernel free the slot. A zombie is not a bug; it is undelivered mail. The bug is a parent that never reads its mail: each unreaped zombie pins a process-table slot, and per wait(2), if that table fills, no new processes can be created on the system.

Which raises the question: what if the parent dies first? The child is now an orphan, and orphans get reparented, adopted by init (PID 1), whose entire job description includes calling wait() in a loop so zombies never accumulate. Since Linux 3.4 there is a refinement: a process can mark itself a subreaper with prctl(PR_SET_CHILD_SUBREAPER), and orphaned descendants then reparent to the nearest living ancestor subreaper instead of falling all the way to PID 1. Session managers and container runtimes use this to keep custody of their subtree's exit statuses.

One last wrinkle, because it bites real systems: PIDs are recycled. You note that some process is PID 4321, it dies, another process starts and gets 4321, and your kill(4321, SIGTERM) murders a stranger. The classic fix was "only signal your own children, and only before reaping." The modern fix (Linux 5.3) is the pidfd: pidfd_open(pid, 0) returns a file descriptor that refers to that specific process instance, not the number. A pidfd never migrates to a recycled PID; you can poll() it to learn the process exited, signal through it with pidfd_send_signal(), and, if it is your child, wait on it with waitid(). Process handles finally behave like file handles.

Signals: a tap on the shoulder, with rules [working]§

Signals are the kernel's way to interrupt a process asynchronously: SIGSEGV when it touches bad memory, SIGCHLD when a child dies, SIGINT for Ctrl-C, SIGTERM/SIGKILL to ask/force it to die. Delivery is two-phase. First the signal is marked pending, which is a bit in the task's pending set. Then, typically as the task transitions back from kernel mode to user mode, the kernel checks the pending set and acts: run the default action (terminate, ignore, stop, dump core), or, if the process installed a handler with sigaction(), hijack the user-mode execution to run the handler function and return to the interrupted code afterward. Two signals refuse negotiation: SIGKILL and SIGSTOP cannot be caught or ignored, so the system always retains a kill switch.

Handlers come with a trap that has broken countless programs. Your handler runs in the middle of whatever the thread was doing, possibly halfway through malloc() updating the heap's internal lists. If the handler itself calls malloc(), it re-enters that half-updated state and corrupts the heap. Same for printf(), which shares stdio buffers with the interrupted code; per signal-safety(7), the second call "will operate on inconsistent data, with unpredictable results." Only async-signal-safe functions may be called from a handler, meaning ones that are reentrant or atomic with respect to signals, like write(2). POSIX publishes the exact list in signal-safety(7). The professional pattern: the handler does almost nothing. It sets a volatile sig_atomic_t flag, or writes one byte to a pipe your event loop watches, and the main program does the real work at a safe point. Treat a signal handler like an interrupt service routine, because that is what it is.

If you have written an ISR on a microcontroller, you already hold this rule and paid for it once. The list of functions you may call is different, and the reason is identical.

The real thing in Linux [working]§

task_struct, grouped by concern [working]§

Everything above lives in one struct: struct task_struct, defined in include/linux/sched.h. One instance per task, roughly 800 lines in v6.12, but legible once you group fields by which ownership question they answer. Field names below are verified against the v6.12 tree:

ConcernFields (v6.12)Notes
Identitypid_t pid; pid_t tgid; char comm[TASK_COMM_LEN];pid is the per-task id (userspace "TID"); tgid is what getpid() returns. comm is the executable name, sans path.
Stateunsigned int __state;The R/S/D/T machinery; Z lives in the exit path.
Schedulingint prio; int static_prio; int normal_prio; unsigned int policy; const struct sched_class *sched_class;The scheduler's hooks, chapter 9's territory.
Memorystruct mm_struct *mm; struct mm_struct *active_mm;The thread test: tasks with the same mm are threads of one process (CLONE_VM). Kernel threads have mm == NULL and borrow active_mm.
Filesstruct files_struct *files;The fd table; shared under CLONE_FILES.
Credentialsconst struct cred __rcu *cred, *real_cred;Every permission check reads cred. Coconut's shim hangs off this; see below.
Signalsstruct signal_struct *signal; struct sighand_struct __rcu *sighand; sigset_t blocked; struct sigpending pending;sighand is the handler table shared under CLONE_SIGHAND.
Familystruct task_struct __rcu *real_parent, *parent; struct list_head children, sibling; struct task_struct *group_leader; struct list_head thread_node;The process tree, as literal linked lists; thread_node chains a thread group together.
Kernel stackvoid *stack;Each task also owns a small kernel-mode stack, separate from its user stack.

Notice what is pointers: mm, files, sighand are references to separately refcounted objects. That is how sharing works: CLONE_VM means "take a reference to my mm_struct instead of copying it." The clone-flags table from Unfolded is, concretely, a list of which pointers get shared versus deep-copied.

The creation, exec, and exit paths [working]§

On x86_64 in v6.12 (arch/x86/entry/syscalls/syscall_64.tbl): clone = 56, fork = 57, vfork = 58, execve = 59, wait4 = 61, pidfd_send_signal = 424, pidfd_open = 434, clone3 = 435. The 64-bit ABI numbering tops out at 462 (mseal); the same file also carries legacy x32-ABI entries at 512-547.

Coconut tie-in [working]§

Coconut OS, currently in spec phase and pre-implementation, is a hard fork of Linux 6.12 LTS that adds a second unit of ownership alongside the process: the agent. The premise: an AI agent is a principal that spans processes (a planner forks tool-runners, which fork shells), so accountability pinned to a single PID is the wrong granularity, the way "one program owns the machine" was the wrong granularity before processes existed.

Concretely, per 04-HLD and 05-LLD:

The design bet is the same one Unix made with the process: make the ownership unit a kernel primitive and the hard questions (who did this, what may it do, what do we reclaim when it dies) get answered by construction rather than by convention.

Lab [working]§

All three labs are short, and all three are worth typing rather than reading. Host is macOS, so run them inside a Linux environment: the book's QEMU guest, or a throwaway container.

sh
docker run --rm -it -v "$PWD":/lab -w /lab gcc:14 bash
# inside, if `ps` is missing:  apt-get update && apt-get install -y procps

Lab 1: a shell in 60 lines [working]§

Predict first: which process runs execvp, the parent or the child? What happens if you type a nonexistent command? Then build it.

c
/* minish.c - fork + execvp + waitpid is a whole shell */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>

#define MAX_ARGS 64

int main(void)
{
    char line[1024];

    for (;;) {
        fputs("minish> ", stdout);
        fflush(stdout);
        if (!fgets(line, sizeof(line), stdin))
            break;                          /* EOF: Ctrl-D */

        line[strcspn(line, "\n")] = '\0';
        if (line[0] == '\0')
            continue;
        if (strcmp(line, "exit") == 0)
            break;

        char *argv[MAX_ARGS];
        int argc = 0;
        for (char *tok = strtok(line, " \t");
             tok && argc < MAX_ARGS - 1;
             tok = strtok(NULL, " \t"))
            argv[argc++] = tok;
        argv[argc] = NULL;

        pid_t pid = fork();
        if (pid < 0) { perror("fork"); continue; }

        if (pid == 0) {                     /* child */
            execvp(argv[0], argv);
            perror("execvp");               /* reached only on failure */
            _exit(127);
        }

        int status;                         /* parent */
        if (waitpid(pid, &status, 0) < 0)
            perror("waitpid");
        else if (WIFEXITED(status) && WEXITSTATUS(status) != 0)
            fprintf(stderr, "[exit %d]\n", WEXITSTATUS(status));
    }
    return 0;
}
sh
gcc -Wall -o minish minish.c && ./minish

Expected:

minish> echo hello from a real shell
hello from a real shell
minish> ls /nonexistent
ls: cannot access '/nonexistent': No such file or directory
[exit 2]
minish> nosuchcmd
execvp: No such file or directory
minish> exit

The child calls execvp; if exec succeeds, the perror line never runs, because the process is now running a different program. The [exit 2] line is the parent reading the child's status out of waitpid, the same mail a zombie holds.

That is a working shell, not a toy version of one. Fork, exec, wait is the load-bearing core of every Unix shell including bash, and most of what bash adds on top of those three calls is convenience. Add one feature to prove you own it: handle cmd > file by opening the file in the child and calling dup2(fd, 1) before execvp. That is the redirection mechanic from the execve section, written out in four lines.

Lab 2: manufacture a zombie [working]§

Predict: after the child exits but before the parent reaps it, what will ps show in STAT, and how much memory will the zombie hold?

c
/* zombie.c */
#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void)
{
    pid_t pid = fork();
    if (pid == 0)
        _exit(0);                 /* child dies instantly */

    printf("child %d exited but is NOT reaped. Check ps now.\n", pid);
    sleep(30);

    waitpid(pid, NULL, 0);        /* reap */
    printf("reaped. Check ps again.\n");
    sleep(30);
    return 0;
}
sh
gcc -Wall -o zombie zombie.c && ./zombie &
sleep 1; ps -o pid,ppid,stat,comm

Expected (during the first 30 seconds):

  PID  PPID STAT COMMAND
   12     1 S    bash
   47    12 S    zombie
   48    47 Z    zombie <defunct>

Z plus <defunct>: terminated, memory already reclaimed, only the exit-status stub remaining. Run ps again after the "reaped" message and PID 48 is gone. waitpid collected the mail and release_task() freed the slot.

Lab 3: watch copy-on-write in /proc [working]§

Predict: right after fork, does the child's 64 MiB buffer show as shared or private dirty memory? After the child overwrites half of it?

c
/* cow.c */
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>

#define SZ (64 * 1024 * 1024)     /* 64 MiB */

int main(void)
{
    char *buf = malloc(SZ);
    memset(buf, 0xAA, SZ);        /* fault every page in before forking */

    pid_t pid = fork();
    if (pid == 0) {
        printf("child %d: phase 1 (nothing written). Measure me.\n", getpid());
        sleep(20);
        memset(buf, 0x55, SZ / 2);   /* write half: forces page copies */
        printf("child: phase 2 (half written). Measure me again.\n");
        sleep(30);
        _exit(0);
    }
    printf("parent %d, child %d\n", getpid(), pid);
    sleep(60);
    return 0;
}

In a second terminal (docker exec -it <container> bash if containerized), sum the child's smaps counters during each phase. Shared_* counts pages also mapped by another process (here, the parent); Private_* counts pages only this process maps:

sh
gcc -Wall -o cow cow.c && ./cow &
# phase 1, within the first 20s:
awk '/^Shared_Dirty/ {s+=$2} /^Private_Dirty/ {p+=$2}
     END {printf "Shared_Dirty %d kB  Private_Dirty %d kB\n", s, p}' \
    /proc/<CHILD_PID>/smaps
# phase 2, after "half written":
#   ...same command again

Expected (approximate, because other mappings add noise):

phase 1:  Shared_Dirty ~65536 kB   Private_Dirty ~small
phase 2:  Shared_Dirty ~32768 kB   Private_Dirty ~32768 kB

Phase 1: the child "has" 64 MiB, but every page of it is the parent's physical memory, shared read-only. Fork copied nothing but page tables. Phase 2: the 32 MiB the child wrote triggered faults and got private copies; the untouched half is still shared. That is copy-on-write, measured. (smaps requires CONFIG_PROC_PAGE_MONITOR; any stock distro kernel has it.)

You now have a way to answer "how much memory does this process actually cost" that survives contact with a real system, which the RSS column does not, because RSS counts shared pages in full for every process that maps them. Point the same awk at any long-running process on the box and read its real private footprint. If you want the sharper version of the same measurement, sum Pss instead: it divides each shared page by the number of processes mapping it, so per-process figures add up without counting the same page twice.

Bridge notes [fundamental]§

You already knowWhat is new at this scale
Microcontroller firmware: one program owns all memory and peripheralsA process is that same "owns the whole machine" illusion, multiplied: the MMU and the kernel fake a private machine per process
ISR save/restore: push registers, run handler, pop, resumeA context switch is the same move between whole tasks; task_struct is where the "pushed registers" of a descheduled task conceptually live
Interrupt vectors dispatching to handlers by numberSignals are software-level vectors: numbered events dispatched to per-process handlers, with the same reentrancy discipline ISRs demand (async-signal-safety)
Compiler symbol tables: one record per name, holding everything known about ittask_struct is the kernel's symbol-table entry for a running program: identity, type ("scheduling class"), storage (mm), linkage (parent, children)
Activation records / call stacks from compiler designEach task carries two stacks: its user stack, plus a small kernel stack (task_struct.stack) for when it traps into the kernel
Structural sharing in immutable data structures (share subtrees, copy on modify)COW fork is structural sharing applied to an entire address space, with the MMU's write-protection faults as the mutation detector

Sources [fundamental]§