Chapter 2: Interrupts & Exceptions§

Authors: Shrey Patel and Jay Patel, Coconut Labs
Book: Think Like an OS. See book/00-INDEX.md

After this chapter you can explain, end to end, how a keypress or an NVMe completion travels from a wire into a driver function; read /proc/interrupts and know what every column and row means; trace real hardware-interrupt entry and exit with ftrace inside a QEMU guest; deliberately crash a program two different ways and name the exact CPU exception vector behind each crash; and explain why interrupt-handler code is forbidden from sleeping, a rule you will keep tripping over for the rest of this book.

This chapter pays off in two different directions. If you are heading for kernel or driver work, the interrupt path is the first code you write that the whole machine has to wait for, and the no-sleeping rule is the constraint every driver design bends around. If you are heading for low-latency work, this is where your jitter lives: a device interrupt can land between any two of your instructions, the handler runs with interrupts disabled on that core, and the timer tick fires whether or not your thread wants it. Pinning threads, isolating cores, and busy-polling a NIC instead of taking its interrupt are all attempts to control what this chapter describes.

The problem [fundamental]§

A CPU core retires instructions on the order of billions per second. A disk finishes a read in tens of microseconds; a network packet arrives whenever it feels like it; a key is pressed maybe five times a second. The CPU and the outside world run on wildly different clocks, and the CPU has no built-in way to know when something outside it has happened.

The obvious answer is to ask. In a loop. "Is the data ready? Is the data ready? Is the data ready?" This is called polling, and it fails in both directions at once. Poll tightly and you burn an entire core doing nothing but asking: millions of wasted checks for every one that says yes. Poll loosely and you add latency: if you check the network card every 10 milliseconds, a packet that arrives right after a check sits there for 10 milliseconds before anyone notices. You can't tune your way out; any polling interval is either too hot or too late for some workload. And with dozens of devices, your kernel's main loop becomes a round-robin of questions with mostly-no answers.

The fix is an inversion of control, the same idea that makes event-driven code work everywhere from GUIs to Node.js: don't call the hardware; let the hardware call you. The device gets a wire (or, later in this chapter, the ability to write a special memory message) that says "I have news." When it fires, the CPU stops what it's doing, mid-program, between two instructions. It saves enough state to resume later and jumps to a function the kernel registered in advance for exactly this event. That mechanism is the interrupt. It is the single reason a kernel can service a thousand devices while spending nearly all of its time running your programs.

There's a sibling mechanism with the same machinery and the opposite trigger. Sometimes the news isn't from outside. It is the CPU itself reporting that the instruction it is currently executing cannot proceed: a divide by zero, a touch of an unmapped address, a privileged operation from unprivileged code. These are exceptions, and they reuse the same save-state-and-jump plumbing. Together, interrupts and exceptions are the only ways control ever enters a kernel while it isn't already running. Every system call, every scheduler decision, every page of memory faulted in on demand: all of it rides on the mechanism in this chapter.

Unfolded [fundamental]§

What actually happens, step by step [fundamental]§

Strip away architecture details and every interrupt or exception follows the same five-step script:

  1. An event fires. A device raises a signal, or an instruction hits a condition it cannot complete.
  2. The CPU finishes (or suspends) the current instruction. Interrupts are only taken at instruction boundaries, so the interrupted program never sees a half-executed instruction.
  3. The CPU saves the minimum resume state. At least the program counter (where to come back to) and the processor flags, pushed onto a stack or stashed into special registers.
  4. The CPU looks up a handler in a table the kernel filled in at boot. The event carries a number; the number indexes the table; the table entry holds the address of a kernel function. This lookup is done by hardware, in nanoseconds, with no software in the path.
  5. The CPU jumps to that handler, switching to privileged (kernel) mode if it wasn't already there. The handler does its job and executes a special return instruction that restores the saved state, and the interrupted program continues, oblivious.

That table in step 4 is the heart of everything. On x86_64 it's called the IDT; on ARM64 it's the vector table. The number in step 4 is called a vector. Registering an interrupt handler (something you'll do in Chapter 12 when we write a driver) ultimately means getting your function reachable from one slot of that table.

One vocabulary distinction to lock in now, because the rest of the chapter leans on it: an interrupt is asynchronous. It comes from outside the instruction stream (a device, a timer, another CPU) and could arrive between any two instructions. An exception is synchronous. It is caused by the instruction being executed right now, and running the same program on the same data reproduces it at the same spot. Hardware routes both through the same table; the kernel treats them very differently, because an exception has a guilty party (the current program) and an interrupt does not.

x86_64: the IDT and the exception zoo [working]§

On x86_64, the lookup table is the Interrupt Descriptor Table (IDT). It has 256 slots, numbered 0 to 255; each slot is a 16-byte gate descriptor holding the handler's address, the stack to use, and privilege rules. The kernel builds the table in memory and tells the CPU where it lives by loading the table's address into a dedicated register (with the lidt instruction). From then on, hardware does the dispatch.

The 256 vectors split into two ranges. Vectors 0 to 31 are architecturally reserved: Intel and AMD define what most of them mean, and they're almost all CPU-detected exceptions. Vectors 32 to 255 are free for the OS, which uses them for device interrupts (assigned via the interrupt controllers we'll meet shortly), inter-processor messages, and software-triggered entries. The ones worth memorizing, because you will see these numbers in crash logs for the rest of your career:

VectorMnemonicNameClassTypical trigger
0#DEDivide ErrorFaultInteger divide by zero (or quotient overflow)
2n/aNMIn/aNon-maskable interrupt: hardware failure, watchdog
3#BPBreakpointTrapint3 instruction, how debuggers stop you
6#UDInvalid OpcodeFaultExecuting garbage or an unsupported instruction
8#DFDouble FaultAbortAn exception raised while delivering another exception
13#GPGeneral ProtectionFaultPrivilege and segment violations, bad register writes: the catch-all
14#PFPage FaultFaultTouching a virtual address with no valid mapping (Chapter 4's whole story)
18#MCMachine CheckAbortThe hardware itself detected corruption

The Linux 6.12 header arch/x86/include/asm/trapnr.h encodes exactly this list as X86_TRAP_DE = 0, X86_TRAP_DF = 8, X86_TRAP_GP = 13, X86_TRAP_PF = 14, and so on. Those are the kernel's names for the architecture's numbers.

The "Class" column is the fault/trap/abort taxonomy, and it matters because it answers the question: after the handler runs, where does execution resume?

ClassReportedSaved return address points atRestartable?Canonical example
FaultBefore the instruction completesThe faulting instruction itselfYes: fix the cause, retry the instructionPage fault (#PF)
TrapAfter the instruction completesThe next instructionYes, continueBreakpoint (#BP)
AbortPossibly impreciseNot reliableNoDouble fault (#DF), machine check (#MC)

Faults are the interesting ones, because "fix the cause and retry" gives the kernel a second chance on any memory access. When your program touches memory that's valid-but-not-yet-loaded, the CPU raises #PF pointing back at the touching instruction, the kernel maps the page, returns, and the instruction re-executes, this time succeeding. Your program never knows. Demand paging, copy-on-write, memory-mapped files: all of them are page-fault handlers exploiting restartability. Traps resume after the instruction, which is right for breakpoints: re-running int3 would loop forever. Aborts mean the machine state can't be trusted; the kernel's job is to fail loudly. A double fault (vector 8) deserves special respect: it means the CPU hit an exception while trying to deliver a previous exception, usually a sign the kernel's own stack or tables are broken. If delivery of the double fault itself fails, the CPU gives up entirely and resets. That's a triple fault, and it's why a badly broken kernel reboots the machine rather than printing anything.

ARM64: one table, four kinds, four origins [working]§

ARM64 (the architecture's own name is AArch64) organizes the same job differently, and the difference is instructive. Instead of 256 numbered slots, ARM64 has a single vector table with 16 entries, whose base address the kernel writes into a system register called VBAR_EL1. Each entry is not a pointer but a 128-byte slot of actual code, enough room for about 32 instructions, typically a stack adjustment and a branch to the real handler.

Why 16? Because ARM64 classifies an incoming event along two axes. First, what kind of event is it? There are four kinds:

Second, where did the CPU come from? There are four origins: the kernel's own exception level using either of the two stack-pointer conventions (called EL1t and EL1h; exception levels are ARM's privilege rings, EL0 for user programs and EL1 for the kernel), a 64-bit user program (EL0), or a 32-bit user program. Four kinds times four origins gives sixteen entries:

VBAR_EL1 ──► ┌────────────────────────────┬─────────┐
             │ Synchronous     from EL1t  │ +0x000  │
             │ IRQ             from EL1t  │ +0x080  │
             │ FIQ             from EL1t  │ +0x100  │
             │ SError          from EL1t  │ +0x180  │
             ├────────────────────────────┼─────────┤
             │ Synchronous     from EL1h  │ +0x200  │  ← kernel faulted
             │ IRQ             from EL1h  │ +0x280  │  ← device interrupted kernel
             │ FIQ             from EL1h  │ +0x300  │
             │ SError          from EL1h  │ +0x380  │
             ├────────────────────────────┼─────────┤
             │ Synchronous  from EL0/64   │ +0x400  │  ← syscall or user fault
             │ IRQ          from EL0/64   │ +0x480  │  ← device interrupted user code
             │ FIQ          from EL0/64   │ +0x500  │
             │ SError       from EL0/64   │ +0x580  │
             ├────────────────────────────┼─────────┤
             │ Synchronous  from EL0/32   │ +0x600  │
             │ IRQ          from EL0/32   │ +0x680  │  (32-bit compat)
             │ FIQ          from EL0/32   │ +0x700  │
             │ SError       from EL0/32   │ +0x780  │
             └────────────────────────────┴─────────┘

Notice what moved where. On x86_64, "which device?" is encoded in the vector number, and there are 256 of them. On ARM64, all device interrupts land on one IRQ entry per origin, and the handler must then ask the interrupt controller "who was it?" And where x86 gives every exception its own vector, ARM64 funnels all synchronous exceptions into one entry and the handler reads a syndrome register (ESR_EL1) that describes exactly what happened and why. Same information, different division of labor between hardware table and software decode.

Interrupt controllers: the routers in the middle [working]§

So far we've pretended devices are wired straight into the CPU. They aren't. With dozens of devices and many CPU cores, you need an arbiter in the middle deciding which events reach which core, in what order, with what priority. That arbiter is the interrupt controller, and each architecture has its own.

On x86, the modern arrangement has two cooperating parts. Each CPU core contains a local APIC (Advanced Programmable Interrupt Controller): it receives interrupt messages aimed at that core, prioritizes them, and injects them with the right vector number. It also generates the most important interrupt in the system: each local APIC contains a programmable timer, and that timer is how the kernel gets its periodic heartbeat (more below). It also sends inter-processor interrupts (IPIs), which are how one core pokes another: "reschedule yourself," "flush your TLB." Separately, the chipset provides one or more IO-APICs: routers that take legacy interrupt lines from devices and forward them, via a configurable redirection table, as messages to some core's local APIC. (Their ancestor, the 8259 PIC, managed 8 lines in the original IBM PC, 15 usable once the PC/AT cascaded a second one, and always exactly one CPU. It survives only as a compatibility relic.)

Modern PCIe devices mostly skip the IO-APIC entirely. With MSI (Message Signaled Interrupts) a device raises an interrupt by performing a memory write to a special address. The "wire" is a message on the same bus the data uses. MSI, first specified in PCI 2.2, gives a device up to 32 vectors; MSI-X (PCI 3.0) extends that to as many as 2048 per device, each independently routable to a different core. This is why your NVMe drive or 100G NIC can have a separate interrupt per queue, each steered to the core that owns that queue, and why MSI interrupts are never shared between devices, eliminating a whole class of "whose interrupt was that?" overhead. If you have ever pinned a process to the same core that handles its NIC receive queue, MSI-X is the hardware reason that works: the queue's interrupt is independently routable, so the data and the notification can be made to land on one core.

 legacy device ──wire──► IO-APIC ──message──► local APIC ──vector──► CPU 0
                                                (per-core)
 NVMe queue 3 ────────MSI-X memory write────► local APIC ──vector──► CPU 3
 CPU 0 ───────────────────IPI───────────────► local APIC ──vector──► CPU 5

On ARM64 the equivalent is the GIC (Generic Interrupt Controller), today usually GICv3. It splits into a distributor (one per system, routes shared interrupts), a redistributor per core (handles that core's private interrupts), and a CPU interface per core (the part the core actually talks to; in GICv3 that happens via system registers rather than memory-mapped I/O, and it can address more than 8 cores, which was GICv2's ceiling). The GIC also names its interrupt categories explicitly:

GIC classStands forScopeTypical use
SGISoftware Generated InterruptCore-to-coreARM's IPIs
PPIPrivate Peripheral InterruptPer-coreEach core's architected timer
SPIShared Peripheral InterruptSystem-wideOrdinary devices (UART, GPU, ...)
LPILocality-specific Peripheral InterruptMessage-basedMSI-style, via the GIC's ITS

Different acronyms, same physics: per-core delivery hardware, a shared router, and a message-based path for high-volume PCIe devices.

The cardinal rule: interrupt context cannot sleep [working]§

When your handler runs in response to a hardware interrupt, the kernel says it runs in hardirq context, and one rule towers over all others: code in hardirq context must never sleep.

"Sleep" has a precise meaning here. Normally, when kernel code needs something it must wait for (a lock someone else holds, a page that must be read from disk, a memory allocation that requires reclaiming) it blocks: the scheduler puts the current thread aside and runs something else. That entire machine assumes there is a current thread to put aside. An interrupt handler has none. It borrowed the CPU from whatever happened to be running: maybe your text editor, maybe the idle loop, maybe another kernel path holding delicate locks. There is no schedulable identity to suspend, and on that CPU further interrupts are typically disabled while the handler runs. If a handler blocks, that core is gone. No scheduler runs on it, and nothing wakes it. Block on a lock held by a thread on this same core and you've deadlocked it permanently.

So in hardirq context you may not: acquire sleeping locks (mutexes), allocate memory in the normal may-block way, do file or disk I/O, copy from user memory (it might page-fault, which might require disk), or call anything that might do any of the above transitively. What remains: read/write your device's registers, grab spinlocks (locks that busy-wait rather than sleep; Chapter 7), and set state for someone else to act on. That last item is not a consolation prize; it's the design.

Top half, bottom half: do less now, more later [working]§

Because handlers can't sleep and shouldn't run long (they're holding up everything on that core, including other interrupts), Linux splits interrupt work in two. The top half is the hardirq handler: acknowledge the device, capture whatever is volatile (grab the data pointer, silence the interrupt), schedule follow-up, return. Microseconds. The bottom half is the follow-up: it runs later, in a friendlier context, and does the real work: processing a network packet through the protocol stack, completing a block I/O request.

Linux has accumulated several bottom-half mechanisms; you'll meet all of them in real drivers:

MechanismRuns inCan sleep?Status in 6.12
SoftirqInterrupt-return path / ksoftirqdNoCore infrastructure; fixed set, not for drivers
TaskletSoftirq contextNoDeprecated; being converted away
BH workqueueSoftirq-like contextNoAdded in 6.9 as the tasklet replacement
WorkqueueKernel worker threadYesDefault for non-urgent deferral
Threaded IRQDedicated per-IRQ threadYes (thread part)Recommended for new drivers

The timer interrupt: the heartbeat [fundamental]§

One interrupt outranks all others in consequence: the timer. Each core's local APIC timer (x86) or architected per-core timer delivered as a PPI (ARM64) fires periodically, at the rate of the kernel's HZ config (selectable as 100, 250, 300, or 1000 per second, defaulting to 250), and its handler is how the kernel reclaims control from whatever is running.

A user program in a tight infinite loop makes no system calls and touches no bad memory: it never voluntarily enters the kernel. Without the timer, that program owns its core forever, and no amount of clever kernel code can do anything about it, because the kernel isn't running. The timer interrupt is the mechanism that makes preemptive multitasking possible at all: every few milliseconds, the hardware yanks control from the running program and hands it to the kernel, which updates time accounting and asks the scheduler, "should someone else run now?" In 6.12 that question is literally a function: the timer path calls update_process_times(), which calls sched_tick(). Chapter 9 lives inside that function. (Modern kernels also run tickless, suppressing the periodic tick on idle or isolated cores to save power, but the principle stands: the scheduler's authority flows from a hardware timer the running program cannot refuse.) That tickless machinery is also what a latency-sensitive shop is buying when it isolates a core. Fewer forced entries into the kernel per second means fewer chances to disturb the thread that must not be disturbed.

The real thing in Linux [working]§

Where this chapter lives in a 6.12-era tree, should you want to walk the code:

Coconut tie-in [working]§

Coconut OS (spec phase, pre-implementation) is a Linux 6.12 LTS fork, so everything above is inherited unchanged. We deliberately do not touch the IDT, vector table, or genirq layer. But three Coconut designs sit directly on top of this chapter's mechanics:

Lab [working]§

Host is macOS, and everything here reads /proc and tracefs, so run inside a Linux environment: the book's QEMU guest (see Chapter 1's lab setup) is ideal; a Docker container works for Lab 1 and Lab 3 (docker run --rm -it ubuntu bash; remember Docker Desktop runs a Linux VM, and that VM's kernel is what you'll observe), but Lab 2 needs tracefs, so use the QEMU guest or add --privileged. You'll want gcc and watch (apt-get update && apt-get install -y gcc procps).

Lab 1: watch interrupts happen [fundamental]§

Predict first: which row of /proc/interrupts will climb fastest while the system is idle? And which will jump when you hammer the disk?

sh
cat /proc/interrupts

Expected: one column of counts per CPU, then the controller type and device name. On an x86 QEMU guest you'll see numbered device rows (virtio devices, often via PCI-MSIX) and named per-CPU rows: LOC (local timer), RES (rescheduling IPI), CAL (function-call IPI), NMI:

           CPU0       CPU1
  1:          9          0   IO-APIC   1-edge      i8042
 27:      12345          0   PCI-MSIX-0000:00:04.0  0-edge  virtio2-req.0
LOC:      84211      79804   Local timer interrupts
RES:       1203       1187   Rescheduling interrupts

Now generate I/O and watch:

sh
watch -n1 'grep -E "LOC|virtio|nvme|ata" /proc/interrupts'
# in a second shell:
dd if=/dev/zero of=/tmp/blast bs=1M count=512 oflag=direct && rm /tmp/blast

Expected: the disk row (virtio-blk/nvme) jumps by thousands during the dd, one interrupt per completed batch of I/O, then goes quiet. LOC climbs steadily regardless, at roughly HZ per second per busy CPU: the heartbeat, beating whether or not anything is happening. If it climbs much slower on an idle CPU, you're watching the tickless mode mentioned above.

Lab 2: trace hardirq entry and exit with ftrace [working]§

Predict first: for each disk interrupt, how many microseconds between handler entry and exit? (Recall the top-half doctrine before answering.)

sh
cd /sys/kernel/tracing            # older systems: /sys/kernel/debug/tracing
echo 1 > events/irq/irq_handler_entry/enable
echo 1 > events/irq/irq_handler_exit/enable
echo 1 > events/irq/softirq_entry/enable
echo 1 > tracing_on
dd if=/dev/zero of=/tmp/blast bs=1M count=64 oflag=direct
echo 0 > tracing_on
head -40 trace

Expected: entry/exit pairs, tagged with the IRQ number and device name matching Lab 1's rows. Watch the pattern of softirq work following hardirq exit:

  <idle>-0    [000] d.h1.  613.204512: irq_handler_entry: irq=27 name=virtio2-req.0
  <idle>-0    [000] d.h1.  613.204518: irq_handler_exit: irq=27 ret=handled
  <idle>-0    [000] ..s1.  613.204521: softirq_entry: vec=4 [action=BLOCK]

Entry-to-exit is single-digit microseconds; the top half really does almost nothing. The BLOCK softirq firing immediately after is the bottom half picking up the real work. The h and s in the flags column are ftrace telling you the context: hardirq and softirq. Disable with echo 0 into the same files when done.

What you just built is the standard instrument for chasing interrupt jitter in production, and it is reading real events on your machine at microsecond resolution. The same event names work on any Linux box you are ever handed.

Lab 3: cause exceptions on purpose [fundamental]§

Predict first: what message will each program die with, and which vector from the x86 table is responsible?

sh
cat > /tmp/div0.c <<'EOF'
int main(void) { volatile int zero = 0; return 1 / zero; }
EOF
cat > /tmp/null.c <<'EOF'
int main(void) { volatile int *p = 0; return *p; }
EOF
gcc -O0 -o /tmp/div0 /tmp/div0.c && gcc -O0 -o /tmp/null /tmp/null.c
/tmp/div0; echo "exit: $?"
/tmp/null; echo "exit: $?"

Expected:

Floating point exception (core dumped)
exit: 136
Segmentation fault (core dumped)
exit: 139

Unfold what happened. div0 executed an integer divide by zero; the CPU raised vector 0 (#DE); the kernel's handler saw it came from user code and delivered the signal SIGFPE (signal 8, and 136 = 128 + 8; "floating point" is a naming fossil, the divide was pure integer). null touched address 0; no mapping exists; vector 14 (#PF); the page-fault handler found no way to fix it and delivered SIGSEGV (signal 11; 139 = 128 + 11). Same mechanism, opposite outcomes from Chapter 4's perspective: a page fault the kernel can satisfy is silently repaired and restarted; one it can't becomes a signal. The kernel usually also logs user faults:

sh
dmesg | tail -2

Expected (if the debug.exception-trace sysctl is enabled, which it commonly is):

null[812]: segfault at 0 ip 0000561b2f60113d sp 00007ffca1b0c4a0 error 4 in null[...]

error 4 is the page-fault error code: a read, from user mode, of a not-present page. That is the hardware's own diagnosis, passed through to the log.

Take a segfault line out of any dmesg now and you can say which vector produced it and what the error code claims the program was doing. That decoding is the difference between "it crashed" and a bug report someone can act on.

Bridge notes [fundamental]§

You've written microcontroller ISRs; the concepts transfer almost one-to-one, and the deltas are exactly the interesting part.

You already know (Cortex-M / MCU)Same idea hereWhat changes at this scale
NVIC + vector table at a base addressIDT (x86) / VBAR_EL1 table (ARM64)Same lookup; hundreds of vectors, per-core controllers (local APIC / GIC redistributor), and message-based MSI-X instead of pins
Vector table = array of handler pointersx86 agrees (gate descriptors)ARM64's entries are 128 bytes of code, not pointers
Hardware auto-stacks r0-r3, r12, LR, PC, xPSRHardware saves a minimal frameKernel software saves the rest into struct pt_regs; there's also a privilege switch and (from user mode) a stack switch on entry
NVIC nests by priority: higher preempts lower, freelyPriorities exist in the APIC/GICLinux runs hardirq handlers with local interrupts disabled, so there is effectively no nesting by default; urgency is handled by keeping top halves tiny, not by preemption depth
ISR sets a flag; main loop does the workTop half / bottom halfThe "main loop" is now a whole scheduler-managed menagerie: softirqs, ksoftirqd, workqueues, per-IRQ threads
One core, one interrupt statePer-CPU everythingEach core has its own timer, its own pending softirqs, its own ksoftirqd/N; IRQ affinity decides which core a device interrupts
"Don't do slow things in an ISR" (advice)"Never sleep in hardirq context" (law)On an RTOS you'd block the loop; here you deadlock a core inside a general-purpose kernel. The rule is enforced by debugging assertions and hard experience
svc for RTOS system callsIdenticalLinux syscalls are the same trick: a deliberate synchronous exception, the subject of Chapter 3

From your compilers side, one connection worth making: exceptions are the hardware's contribution to the illusion your compiler also maintains, which is precise, restartable state. A fault delivers the address of the exact instruction, with all prior instructions retired and none after, which is what lets a page-fault handler splice disk I/O into the middle of a mov without the program noticing. Out-of-order CPUs spend enormous silicon preserving that lie; Chapter 4 collects the payoff.

Three things to go do with this. Read /proc/interrupts on any Linux machine you have access to and say which core is absorbing which device. Take a piece of work you have written and decide out loud whether it belongs in a top half or a bottom half. And when you next see a kernel warning about sleeping in atomic context, you will already know what the machine was complaining about. Chapter 3 hands you the memory hardware that those page faults are walking.

Sources [working]§