Chapter 7: The Syscall Boundary§
After this chapter you can invoke a Linux system call from raw assembly with no libc, read an strace log and account for every line in it, look up any syscall's number and entry point in the kernel source tree, explain why gettimeofday() usually costs about as much as a function call, and explain why a syscall number, once shipped, can never be reused. You will also see exactly where Coconut OS's agent_* syscall family (numbers 472-479) plugs into this machinery, and why those specific numbers were chosen.
This is the boundary two different careers meet at. If you are heading for kernel work, everything you will ever write sits on one side of it and every user of your code sits on the other, and the rules for crossing are the rules you will spend years obeying. If you are heading for latency-sensitive work, this chapter is where you learn to see a mode switch in a profile: the round trip costs real time, strace shows you exactly how many of them a program makes, and the vDSO section explains why the one call your hot loop makes most often is not a syscall at all. Both readers leave with the same skill, which is reading a program's behavior as a list of boundary crossings.
The problem [fundamental]§
Your program does not own the machine. It shares the CPU, the memory, the disk, and the network card with every other program, and the kernel is the referee. Chapter 5 established the hardware side of that deal: the CPU runs user code in an unprivileged mode where the dangerous instructions (touching device registers, rewriting page tables, halting the processor) fault.
That creates an immediate, practical problem. Your program needs those dangerous operations constantly. Reading a file, allocating memory, printing to a terminal, opening a socket: all of it ultimately requires privileged work. So unprivileged code must have some way to ask the kernel to do privileged work on its behalf.
The obvious idea is to call a kernel function the way you would call a library. That is exactly what must not exist. A function call transfers control to an address the caller chooses. If user code could jump to arbitrary kernel addresses with privileges raised, it could jump to the middle of a function, skip a permission check, and own the machine. The 30,000 bugs that idea produces are the entire history of computer security compressed into one design mistake.
What's needed instead is a doorbell: a single hardware-enforced entry point whose address the kernel chose in advance, where privileges rise only at that exact address, and where the kernel inspects the request ("syscall number 1, these six argument registers") before doing anything. That mechanism is the system call. The line it draws is the most rigid interface in the system: everything above it can change tomorrow; everything that crosses it is frozen forever.
Unfolded [fundamental]§
A doorbell, not a function call [fundamental]§
A system call (syscall) is a request from a user program to the kernel, made through a special CPU instruction that simultaneously (1) switches the CPU into privileged mode and (2) jumps to one fixed address that the kernel installed at boot. The user program cannot pick the destination. It can only pick which request it is making, by placing a number in a register before ringing the doorbell.
Three terms, defined once:
- Ring 3 / ring 0: x86's names for unprivileged and privileged CPU modes. User programs run in ring 3; the kernel runs in ring 0. (ARM's equivalents are EL0 and EL1, exception levels 0 and 1.)
- MSR: a model-specific register, a named configuration register inside the x86 CPU, readable and writable only from ring 0 with the special
rdmsr/wrmsrinstructions. MSRs are how the kernel configures CPU behavior. - ABI: application binary interface, a contract about registers, numbers, and memory layout that lets separately-compiled machine code interoperate. Calling conventions are ABI. Syscall conventions are ABI.
The doorbell on 64-bit x86 is a two-byte instruction literally named syscall. Where does it jump? At boot, the kernel writes the address of its syscall entry routine into an MSR called MSR_LSTAR (Intel's manuals call it IA32_LSTAR; "LSTAR" is Long-mode System-call TARget). In Linux 6.12 you can watch it happen. syscall_init() in arch/x86/kernel/cpu/common.c calls idt_syscall_init() (unless FRED, Intel's redesigned event-delivery hardware, is enabled), which contains:
wrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);
From that moment on, every syscall instruction executed by any ring-3 program lands at entry_SYSCALL_64, an assembly routine in the kernel. User code cannot read or change MSR_LSTAR, because wrmsr faults in ring 3. One door, kernel-chosen, hardware-enforced.
What the syscall instruction actually does [working]§
The syscall instruction is deliberately minimal, which is what makes it fast. Per Intel's specification, here is everything it does, all in the one instruction:
rcx := rip. Save the address of the next user instruction into registerrcx, so the kernel knows where to return.r11 := rflags. Save the CPU flags register intor11.rflags := rflags AND NOT(IA32_FMASK). Clear flag bits the kernel asked to have cleared (notably the interrupt-enable flag).rip := IA32_LSTAR. Jump to the kernel's registered entry point.CPL := 0. Raise privilege to ring 0.
Notice what it does not do: it does not switch stacks, does not save the other registers, does not touch memory at all. All of that bookkeeping is the kernel's job, in software, at entry_SYSCALL_64. The hardware provides the minimum unforgeable step: privilege up, jump to the fixed address. Nothing else.
Here is the full round trip for a write():
user space (ring 3) kernel space (ring 0)
─────────────────── ─────────────────────
printf("hi")
└─ glibc write() wrapper
mov rax, 1 ; syscall # 1 = write
mov rdi, 1 ; arg1: fd (stdout)
mov rsi, buf ; arg2: buffer
mov rdx, 3 ; arg3: count
syscall ───────────────────────► entry_SYSCALL_64 arch/x86/entry/entry_64.S
CPU, in hardware: │ switch to kernel stack
rcx := rip (return addr) │ spill user registers into a
r11 := rflags │ struct pt_regs
rip := MSR_LSTAR ▼
ring 3 → ring 0 do_syscall_64(regs, nr) arch/x86/entry/common.c
│ bounds-check nr < NR_syscalls
│ index the syscall table
▼
sys_write → ksys_write fs/read_write.c
│ ... actual work ...
│ rax := 3 (bytes written)
◄──────────────────── sysretq ──────┘ restore user registers
execution resumes at the address ring 0 → ring 3
saved in rcx, result in rax
The return instruction, sysretq, is the mirror image: it drops back to ring 3, restores rflags from r11, and jumps to the address in rcx. Round trip, on modern hardware, on the order of a hundred nanoseconds before the kernel does any actual work. Cheap, but still tens of times costlier than a plain function call, which is why the vDSO (below) exists.
The register contract, precisely [working]§
The x86_64 Linux syscall ABI, verified against the syscall(2) man page and the kernel's own entry code:
| Role | Register |
|---|---|
| Syscall number | rax |
| Argument 1 | rdi |
| Argument 2 | rsi |
| Argument 3 | rdx |
| Argument 4 | r10 |
| Argument 5 | r8 |
| Argument 6 | r9 |
| Return value | rax |
| Clobbered by the instruction | rcx, r11 |
If you took a compilers course, one entry should look wrong. The System V AMD64 function calling convention passes the fourth argument in rcx. The syscall convention uses r10 instead. Now you know why: the syscall instruction destroys rcx. The hardware commandeers it for the return address (step 1 above), before the kernel ever runs, so a fourth argument in rcx would be gone on arrival. The ABI reroutes argument 4 through r10, which the function-call convention doesn't use for arguments. The kernel's entry code documents this in a comment above entry_SYSCALL_64, where register r10 carries "arg3 (needs to be moved to rcx to conform to C ABI)." The entry path moves r10 into rcx before calling the C implementation, so ordinary C functions can serve as syscall bodies. r11 is clobbered for the same reason: the hardware uses it to stash rflags.
There are at most six arguments. No syscall takes more, ever. Function calls spill extra arguments onto the stack; syscalls can't, because at the moment of entry the kernel doesn't yet trust anything the user stack says. Syscalls that conceptually need more than six parameters take a pointer to a struct instead. Remember that one; Coconut's agent_spawn does exactly that.
Return values and errno. The kernel returns results in rax. Failure is a negative number: -EINVAL is -22, -ENOENT is -2, and so on. The errno you know from C is a libc fiction layered on top: the glibc wrapper checks whether rax came back in the small negative range that can only mean an error (errno values stay below 4096, so -4095 through -1 is unambiguous), and if so, negates it into the thread-local errno variable and returns -1 to your code. The kernel itself knows nothing about a variable called errno.
ARM64: same idea, different doorbell [working]§
On 64-bit ARM the instruction is svc #0, short for supervisor call. The ABI shape is identical, only the register names change: syscall number in w8 (the 32-bit view of register x8), arguments 1-6 in x0 through x5, return value in x0.
The plumbing differs in one instructive way. x86's syscall jumps to a single dedicated address (MSR_LSTAR). ARM64 has no syscall-specific door: svc raises a general synchronous exception, and the CPU vectors through the exception table whose base address the kernel programmed into the system register VBAR_EL1 (Vector Base Address Register). The kernel's exception handler then reads the syndrome register to learn what kind of exception occurred. In Linux 6.12, arch/arm64/kernel/entry-common.c shows the dispatch: el0t_64_sync_handler() (the handler for synchronous exceptions arriving from EL0) switches on the exception class, and the case ESR_ELx_EC_SVC64, meaning "this was an svc from 64-bit userspace," routes to el0_svc(), which calls do_el0_svc() and thence into the same kind of table lookup x86 does. Same doorbell concept, implemented as a special case of the general exception machinery rather than a dedicated fast path.
If you did microcontroller coursework on Cortex-M parts, you have already used this exact mechanism: the SVC instruction that RTOSes use for supervisor entry is the same architectural idea, and VBAR_EL1 is the grown-up sibling of the vector table you placed at address 0.
The syscall table: numbers to functions [working]§
Inside do_syscall_64, the number in rax becomes an index into an array of function pointers. The kernel bounds-checks it first. arch/x86/entry/common.c in 6.12 reads, abridged:
if (likely(unr < NR_syscalls)) {
unr = array_index_nospec(unr, NR_syscalls);
regs->ax = x64_sys_call(regs, unr);
return true;
}
(array_index_nospec clamps the index even under speculative execution. That is a Spectre countermeasure: an attacker must not be able to speculatively index past the table either.) An out-of-range number gets -ENOSYS: "no such syscall."
The table itself is not written in C. It's generated at build time from a plain text file, arch/x86/entry/syscalls/syscall_64.tbl. The format is one line per syscall, <number> <abi> <name> <entry point>:
0 common read sys_read
1 common write sys_write
...
462 common mseal sys_mseal
In Linux 6.12, 462 (mseal) is the highest assigned x86_64 number. (The file also contains a fossil: numbers 512-547 belong to "x32", a mostly-dead ABI for 32-bit pointers on 64-bit CPUs. Ignore it, but expect to see it when you tail the file in the lab.) Architectures newer than x86 don't each keep their own copy; since the 6.11/6.12 era they share a common table, scripts/syscall.tbl, which also topped out at 462 in 6.12.
The entry points named in the table are defined with a macro family. Here is the entire top-level implementation of write in 6.12, from fs/read_write.c:
SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
size_t, count)
{
return ksys_write(fd, buf, count);
}
SYSCALL_DEFINEn(name, type1, arg1, ...), where n is the argument count from 0 through 6, expands into the function sys_write plus metadata and glue (argument-width sanitization, tracing hooks). The __user annotation on the pointer is a promise-marker meaning "this address came from userspace; never dereference it directly." Static checkers enforce that such pointers only flow through copying functions that validate them. That validation discipline is Chapter 8's whole subject.
The vDSO: the syscall that never happens [working]§
Some syscalls are so hot that even a hundred-nanosecond boundary crossing is too expensive. The classic examples are gettimeofday() and clock_gettime(). Event loops, loggers, and profilers call them millions of times per second, and any code that timestamps each item it handles calls them once per item.
Linux's answer is the vDSO, the virtual dynamic shared object. It is a small, kernel-provided shared library that the kernel maps into the address space of every process automatically. Your program finds it not on disk but via the auxiliary vector (the kernel-provided key-value list every process receives at startup), under the tag AT_SYSINFO_EHDR. On x86_64 in 6.12 it exports __vdso_clock_gettime, __vdso_getcpu, __vdso_gettimeofday, __vdso_time, __vdso_clock_getres, and __vdso_getrandom, new in 6.11 (plus __vdso_sgx_enter_enclave on SGX-enabled builds). The vdso(7) man page still lists only the first four; the linker script arch/x86/entry/vdso/vdso.lds.S is the ground truth.
The trick: for these specific queries, the kernel pre-publishes the answer. It maintains a page of memory, mapped read-only into every process, containing the current clock readings and the conversion factors for the CPU's cycle counter. __vdso_clock_gettime runs entirely in ring 3: read the shared page, read the cycle counter, multiply, add, return. In the man page's words, what would have been a syscall becomes "a normal function call and a few memory accesses." No syscall instruction, no mode switch, no kernel entry. strace won't even see it, and that is a diagnostic fact worth memorizing: time calls missing from an strace log are the vDSO working as intended. (For clock IDs the shared page doesn't cover, the vDSO function falls back to the real syscall.)
The practical consequence: whether a timestamp costs you a function call or a mode switch depends on which clock ID you ask for. CLOCK_MONOTONIC and CLOCK_REALTIME are on the fast path; a clock the shared page does not cover is not, and the difference does not show up in your source code at all. If your loop timestamps every item and the timing is worse than you expect, the vDSO fallback is the first thing to check, and strace is how you check it, because the calls that fall back are the ones that appear in the log.
The vDSO is the exception that proves the boundary's rule: the only way to make a syscall free is to arrange, ahead of time, for it not to be a syscall at all.
Watching and filtering the boundary [working]§
Because every request crosses one narrow doorway, the doorway is the perfect observation point.
strace attaches to a process (via the ptrace(2) debugging facility) and records every syscall with decoded arguments and return values. It is the single highest-value debugging tool this book will teach. strace -f follows child processes; strace -e trace=%file filters to file-related calls; strace -c prints a count/time summary instead of a log. What a program says it does is marketing; its strace log is sworn testimony.
seccomp turns the observation point into a checkpoint. Via seccomp(2) with SECCOMP_SET_MODE_FILTER, a process installs a small BPF filter program that the kernel runs on every subsequent syscall, before dispatch. The filter sees a seccomp_data struct (syscall number, architecture, instruction pointer, and the six raw argument registers) and returns a verdict: SECCOMP_RET_ALLOW, SECCOMP_RET_ERRNO (fail it with a chosen error), SECCOMP_RET_KILL_PROCESS, and several others. Container runtimes wrap untrusted workloads in exactly this. The Docker container you'll use for this chapter's lab is itself confined by a seccomp profile, so you will be studying the boundary from inside a fence built on it. Chapter 19 treats sandboxing in full.
Numbers are forever [working]§
Here is the fact that makes syscall design feel different from every other kind of API design: a syscall number, once released in a stable kernel, is permanent. Compiled binaries embed the number, not the name: mov rax, 1. Reuse number 462 for something new and every existing binary that calls mseal silently starts invoking your new thing. You didn't change an API; you changed the meaning of existing machine code, everywhere, retroactively.
The kernel's own documentation for syscall authors states it plainly: a new system call "forms part of the API of the kernel, and has to be supported indefinitely." Even a syscall that turns out to be a mistake is never renumbered. Its slot is implemented by sys_ni_syscall ("not implemented") returning -ENOSYS, forever.
This is one instance of Linux's most famous rule. On December 23, 2012, after a media-subsystem commit changed a returned error code and broke PulseAudio, Linus Torvalds wrote to the subsystem maintainer, Mauro Carvalho Chehab: "If a change results in user programs breaking, it's a bug in the kernel." Then, in capitals that have since become the community's shorthand for the entire policy: "WE DO NOT BREAK USERSPACE!" Then: "How hard is this rule to understand?" The kernel-internal API churns constantly; the syscall boundary is the wall where churn stops.
The table's growth is correspondingly slow and append-only. Linux 6.12 (November 2024, a longterm-support release maintained into 2028) tops out at 462. As of this writing (mid-2026), the upstream development tree has reached 471. The intervening nine slots went to the *xattrat family (463-466), open_tree_attr (467), file_getattr/file_setattr (468-469), listns (470), and rseq_slice_yield (471). Roughly nine numbers in two years. Every one of them is now permanent.
The real thing in Linux [working]§
The 6.12-era map of everything this chapter described:
| What | Where (Linux 6.12) |
|---|---|
x86_64 entry stub (entry_SYSCALL_64) | arch/x86/entry/entry_64.S |
MSR_LSTAR programmed at boot (idt_syscall_init(), via syscall_init()) | arch/x86/kernel/cpu/common.c |
C-side dispatch (do_syscall_64, bounds check) | arch/x86/entry/common.c |
| x86_64 syscall number table | arch/x86/entry/syscalls/syscall_64.tbl |
| Shared table for newer architectures (incl. arm64) | scripts/syscall.tbl |
ARM64 exception dispatch (el0t_64_sync_handler → el0_svc → do_el0_svc) | arch/arm64/kernel/entry-common.c |
write implementation (SYSCALL_DEFINE3) | fs/read_write.c |
Syscall prototypes (asmlinkage long sys_*) | include/linux/syscalls.h |
-ENOSYS stubs for unimplemented calls | kernel/sys_ni.c |
| vDSO sources | arch/x86/entry/vdso/ |
The comment above entry_SYSCALL_64 in entry_64.S is the ABI's primary source, better than any tutorial: "64-bit SYSCALL saves rip to rcx, clears rflags.RF, then saves rflags to r11, then loads new ss, cs, and rip from previously programmed MSRs," followed by the register-by-register contract. When in doubt about the ABI, read that comment, not a blog post.
Adding a brand-new syscall, per the kernel's Documentation/process/adding-syscalls.rst, touches a fixed set of files. For an x86_64-first syscall named xyzzy:
- Implement it as
SYSCALL_DEFINEn(xyzzy, ...)in an appropriatekernel/or subsystem file, behind aKconfigoption. - Declare
asmlinkage long sys_xyzzy(...);ininclude/linux/syscalls.h. - Add
COND_SYSCALL(xyzzy);inkernel/sys_ni.cso configs that compile it out still link. The slot degrades to-ENOSYSinstead of a link error. - Claim the next free number in
arch/x86/entry/syscalls/syscall_64.tbl:463 common xyzzy sys_xyzzy(463 being next after 6.12's 462). - Wire other architectures. For modern ones that is one line in the shared
scripts/syscall.tbl. - Add a self-test and man-page text; on the public list, expect the API-design review to be the hard part, precisely because of the forever rule.
One modernity note: 6.12 also contains early support for FRED, Intel's redesigned event-delivery hardware; on FRED systems the entry plumbing differs (syscall_init() branches on it), but the ABI contract is identical: numbers, registers, return convention. The contract outlives the plumbing.
Coconut tie-in [working]§
Coconut OS, our Linux 6.12 LTS fork and currently in spec phase, makes AI agents first-class kernel citizens. The front door for that is a new syscall family: agent_* at numbers 472-479 (design in 04-HLD; ABI details in 05-LLD). Everything about the range placement follows directly from this chapter's mechanics:
- Why 472? 6.12 ends at 462, but upstream keeps allocating, and the mainline table has already reached 471 in mid-2026. Squatting on 463 would have collided with upstream's
setxattratwithin months, and a collision on a syscall number is unfixable-by-definition once binaries exist: same number, two meanings. 472 sits just past upstream's current frontier, and the spec reserves 480-487 as a fallback block in case upstream reaches 472 before our numbers freeze at Gate 1. - What's wired: 472
agent_spawnand 473agent_attesthave real entry points; 474-479 are reserved stubs returning-ENOSYS, exactly thesys_ni_syscallpattern upstream uses, claiming the numbers now so nothing else grows into them. - The six-argument ceiling in practice:
agent_spawntakes a pointer to a versioned parameter struct rather than a long argument list, the standard idiom for syscalls whose parameter set will grow (compare upstream'sclone3). - The forever rule, inherited: a hard fork doesn't escape ABI gravity. It doubles it. Once Coconut ships binaries, 472-479 are permanent for us; and because upstream doesn't know about our range, every future LTS rebase must re-verify that upstream hasn't allocated into it. That check is cheap, the same
awkone-liner you'll run in the lab, and it runs in our GitLab CI alongside the kunit-coconut QEMU boot gate on every push.
Lab [working]§
Host is macOS, so everything runs inside a Linux container (a QEMU guest works identically). Note the meta-point: Docker confines this container with a seccomp filter, so the lab studies the boundary from inside one.
docker run --rm -it --platform linux/amd64 -v "$PWD":/lab -w /lab debian:bookworm bash
# then, inside the container:
apt-get update && apt-get install -y gcc strace curl
(If strace later fails with a permissions error, restart the container with --cap-add=SYS_PTRACE. On Apple Silicon the --platform linux/amd64 flag gets you an emulated x86_64 userland, which is what this chapter's ABI is about.)
Lab 1: write() with no libc [working]§
Everything libc's write() does at the boundary, by hand. Create raw_write.c:
/* raw_write.c - no libc, no headers. Just the ABI. */
void _start(void)
{
static const char msg[] = "hello from the raw ABI\n";
long ret;
/* write(1, msg, 23) - syscall number 1 */
__asm__ volatile ("syscall"
: "=a" (ret) /* rax out: return value */
: "a" (1L), /* rax in: syscall 1 = write */
"D" (1L), /* rdi: fd 1 (stdout) */
"S" (msg), /* rsi: buffer */
"d" (sizeof(msg) - 1) /* rdx: count */
: "rcx", "r11", "memory");/* what syscall clobbers */
/* exit(0) - syscall 60. Without this we crash: _start was
jumped to, not called, so "return" has nowhere to go. */
__asm__ volatile ("syscall"
: : "a" (60L), "D" (0L)
: "rcx", "r11", "memory");
}
The constraint letters are GCC's register names: "a"=rax, "D"=rdi, "S"=rsi, "d"=rdx. The clobber list declares rcx and r11 dead across the instruction, and you now know exactly why those two.
gcc -nostdlib -static -o raw_write raw_write.c
./raw_write
strace ./raw_write
Expected:
hello from the raw ABI
and from strace, a complete process lifetime in three lines:
execve("./raw_write", ["./raw_write"], 0x7ffd...) = 0
write(1, "hello from the raw ABI\n", 23) = 23
exit(0) = ?
+++ exited with 0 +++
Three syscalls, and you only wrote two of them. execve was made by the shell on your program's behalf; it is how the process came to exist at all. There is no smaller Linux program lifecycle than this.
You just made a system call with no library, no headers, and no runtime between you and the kernel. Every abstraction you use for the rest of your career sits on top of the four instructions you wrote by hand.
Lab 2: account for every syscall in hello world [working]§
Now the same experiment with libc. Create hello.c containing #include <stdio.h> and int main(void){ printf("hello\n"); return 0; }.
gcc -o hello hello.c
strace -c ./hello # summary first
strace -o hello.log ./hello && wc -l hello.log
Expected: on the order of sixty syscalls, not three (the exact list varies with the glibc version; bookworm's glibc 2.36 differs from others in the details, so treat the following as the shape, not gospel). Now read hello.log and account for every line. It decomposes into five phases:
| Phase | Typical syscalls |
|---|---|
| Process creation | execve |
| Dynamic linker finds libc | openat("/etc/ld.so.cache"), openat(".../libc.so.6"), read, pread64, newfstatat, mmap (several, mapping libc's segments), mprotect, close |
| Runtime setup | brk, arch_prctl, set_tid_address, set_robust_list, rseq, prlimit64, getrandom, munmap |
| Your actual program | write(1, "hello\n", 6), one line |
| Shutdown | exit_group(0) |
The gap between 3 and ~60 is the dynamic-linking and libc-startup machinery, all of it crossing the boundary you now understand line by line. Also note what's absent: add a clock_gettime or gettimeofday call to hello.c, rebuild, re-strace. On this ABI it usually never appears in the log. That silence is the vDSO.
Try the same strace -c on something bigger while you have the container open: strace -c ls /usr, or strace -c curl -s example.com >/dev/null. The summary column tells you which syscall a program spends its life in, and that number is often the answer to "why is this slow" before you have opened a profiler.
Lab 3: interrogate the 6.12 table [working]§
Use your 6.12 source tree if you have one from Part 1; otherwise fetch just the table:
curl -sO https://raw.githubusercontent.com/torvalds/linux/v6.12/arch/x86/entry/syscalls/syscall_64.tbl
grep -E '^(0|1|60|462)\b' syscall_64.tbl
Expected:
0 common read sys_read
1 common write sys_write
60 common exit sys_exit
462 common mseal sys_mseal
There are Lab 1's two magic numbers, 1 and 60, in the kernel's own ledger. Now find the true top of the table:
tail -n 3 syscall_64.tbl
awk '$2 == "common" { n = $1; name = $3 } END { print n, name }' syscall_64.tbl
Expected: tail surprises you with numbers in the 540s. That is the legacy x32 block (512-547), not real x86_64 allocations. The awk line gives the honest answer:
462 mseal
That one-liner is, verbatim, the collision check Coconut's CI runs against each upstream rebase: if it ever prints 472 or higher, our agent_* range has been invaded and the fallback block (480-487) activates. Run it against the mainline table (swap v6.12 for master in the URL) and you'll see upstream's frontier creeping toward us: 471 as of mid-2026.
Three labs, and you can now do the whole loop: write a syscall by hand, watch any program's crossings from outside it, and look up the number and entry point of anything you see. Point strace at something you wrote this month. Whatever it does that you did not expect is at the boundary, and now you can read it.
Bridge notes [fundamental]§
You already know calling conventions from compiler coursework and vector tables from microcontroller coursework. The syscall boundary is those two ideas fused, with a privilege gate added. Side by side:
| C function call (System V AMD64) | Linux syscall (x86_64) | |
|---|---|---|
| What selects the target | Symbol → address, resolved at link/load time | Number in rax, fixed forever |
| Who chooses the destination address | The caller (any address) | The kernel (MSR_LSTAR, written at boot) |
| Argument registers | rdi rsi rdx rcx r8 r9, then stack | rdi rsi rdx r10 r8 r9, six max, never stack |
| Transfer instruction | call (return address pushed on stack) | syscall (return address into rcx, flags into r11) |
| Return instruction | ret (pops the stack) | sysretq (drops privilege, jumps to rcx) |
| Return value | rax | rax; negative errno on failure |
| Privilege change | None | Ring 3 → ring 0 → ring 3 |
| Trust model | Callee trusts caller's stack and pointers | Kernel trusts nothing; validates every argument |
| Evolvability | Recompile and everything can change | Append-only, forever |
What you already know → what's new at this scale:
- Calling conventions are arbitrary but sacred → same here, plus a hardware constraint you never had: the transfer instruction itself eats two registers (
rcx,r11), and the convention had to route around it. ABI design bends to silicon. - The stack carries overflow arguments → not here. The kernel won't touch the caller's stack at entry because the caller's stack is untrusted input. Six registers or a pointer to a struct. Nothing else.
- Cortex-M vector table +
SVCinstruction →VBAR_EL1+svc #0is the same architecture grown up;MSR_LSTARis the x86 flavor with exactly one vector. What's new is scale of consequence: your microcontroller's SVC handler served one firmware image you could reflash; this doorway serves every binary ever compiled for Linux, which is why its numbering is append-only and eternal. - Linkers resolve names late → the syscall table resolves numbers never. There is no dynamic linker between userspace and kernel; the number burned into the binary is the contract. That is the deep reason "we do not break userspace" is enforceable at all: the interface is so narrow, and so rigid, that compatibility is a property you can actually audit: one text file, one number column, append-only.
Sources§
- https://man7.org/linux/man-pages/man2/syscall.2.html: x86_64 and arm64 register conventions (rax/rdi-rsi-rdx-r10-r8-r9; svc #0, w8, x0-x5).
- https://www.felixcloutier.com/x86/syscall: SYSCALL instruction semantics: rcx := rip, r11 := rflags, IA32_FMASK, target from IA32_LSTAR, CPL 0.
- https://raw.githubusercontent.com/torvalds/linux/v6.12/arch/x86/entry/entry_64.S: entry_SYSCALL_64 comment block (register contract, r10→rcx note), call to do_syscall_64.
- https://raw.githubusercontent.com/torvalds/linux/v6.12/arch/x86/kernel/cpu/common.c: idt_syscall_init(), called from syscall_init() on non-FRED CPUs: wrmsrl(MSR_LSTAR, entry_SYSCALL_64).
- https://raw.githubusercontent.com/torvalds/linux/v6.12/arch/x86/entry/vdso/vdso.lds.S: the 6.12 x86-64 vDSO export list (clock_gettime, getcpu, gettimeofday, time, clock_getres, getrandom, sgx_enter_enclave).
- https://raw.githubusercontent.com/torvalds/linux/v6.12/arch/x86/entry/common.c: do_syscall_64 dispatch, NR_syscalls bounds check, array_index_nospec.
- https://raw.githubusercontent.com/torvalds/linux/v6.12/arch/x86/entry/syscalls/syscall_64.tbl: 6.12 table: 0 read, 1 write, highest common entry 462 mseal; file format header.
- https://raw.githubusercontent.com/torvalds/linux/master/arch/x86/entry/syscalls/syscall_64.tbl: mainline table as of mid-2026: 463-471 (setxattrat … listns, rseq_slice_yield); x32 block 512-547.
- https://raw.githubusercontent.com/torvalds/linux/v6.12/scripts/syscall.tbl: shared syscall table for newer architectures; also tops at 462 in 6.12.
- https://raw.githubusercontent.com/torvalds/linux/v6.12/arch/arm64/kernel/entry-common.c: el0t_64_sync_handler, ESR_ELx_EC_SVC64 case, el0_svc → do_el0_svc.
- https://raw.githubusercontent.com/torvalds/linux/v6.12/fs/read_write.c: SYSCALL_DEFINE3(write, ...) → ksys_write.
- https://man7.org/linux/man-pages/man7/vdso.7.html: vDSO mechanism, AT_SYSINFO_EHDR, the four x86_64 __vdso_* symbols.
- https://www.kernel.org/doc/html/v6.12/process/adding-syscalls.html: steps and files for adding a syscall; "supported indefinitely" permanence note.
- https://man7.org/linux/man-pages/man2/seccomp.2.html: SECCOMP_SET_MODE_FILTER, seccomp_data fields, SECCOMP_RET_* actions.
- https://man7.org/linux/man-pages/man1/strace.1.html: strace behavior and -f / -e trace= / -c options; ptrace(2) underpinning.
- https://lkml.iu.edu/hypermail/linux/kernel/1212.2/03058.html: Linus Torvalds to Mauro Carvalho Chehab, 2012-12-23: "WE DO NOT BREAK USERSPACE!" and surrounding quotes.
- https://developer.arm.com/documentation/ddi0595/2021-06/AArch64-Registers/VBAR-EL1--Vector-Base-Address-Register--EL1-: VBAR_EL1 as the EL1 exception vector base.
- https://www.kernel.org/category/releases.html: 6.12 longterm status, projected EOL December 2028.