Chapter 9: Scheduling§
After this chapter you can explain what happens in the microseconds between "this task's time is up" and "a different task is running," read the actual EEVDF picker in the 6.12 tree without flinching, and predict, before you run it, what a SCHED_FIFO spinner does to everything else on the machine. You will also know why Linux threw away two schedulers that worked, what the third one fixes, and where Coconut OS's planned SCHED_AGENT class slots into that lineage.
Of the four chapters in this part, this is the one whose payoff is most immediate on both career paths. For kernel work, __schedule() is the function every context switch on the machine passes through, and reading it is a rite of passage. For latency-sensitive work, this chapter is about the number you actually care about, which is not how much CPU your thread gets but how long it waits between becoming runnable and running. That gap has a name here (delay), a cause (the eligibility and deadline rules), a measurement (perf sched latency, in lab 3), and a set of levers with documented semantics. It also has three traps that bite people who reach for SCHED_FIFO as a fix: FIFO does not time-slice, the kernel keeps 5% of the machine away from you on purpose, and moving a task between cores can cost more in cold caches than it saves in queueing. All three are in this chapter, with the experiment that shows each one.
The problem [fundamental]§
Your laptop has 8 CPU cores and, right now, a few hundred processes. Most are asleep waiting for something: a network packet, a keypress, a timer. But at any moment, more tasks may want to run than there are cores to run them. Something has to decide who runs, for how long, and who waits. That something is the scheduler, and every design choice it makes is a trade among three goals that pull against each other:
- Fairness. If two tasks want the CPU equally, each should get half. If one is twice as important, it should get twice the share. Without this, a runaway loop silently steals the machine from everything else.
- Latency. When a sleeping task wakes up (you pressed a key, a packet arrived), how long until it actually runs? A music player that gets its fair 10% of the CPU, but delivered as one big chunk every second, still stutters. Latency is about when you run, not how much.
- Throughput. Every switch between tasks costs time that no task benefits from. Switching rarely maximizes useful work done per second; switching often minimizes waiting. You cannot have both.
A batch server wants throughput. A game wants latency. A multi-tenant box wants fairness. A general-purpose kernel has to serve all three at once, with no advance knowledge of what any task is about to do. That impossible brief is why Linux is on its third general-purpose scheduler, and why the history in this chapter is a history of moving intelligence out of heuristics and into arithmetic.
Unfolded [fundamental]§
The tick: how the kernel gets control back [fundamental]§
First, a mechanical question that every scheduler design depends on: if a task is running a tight infinite loop and never asks the kernel for anything, how does the kernel ever get the CPU back to make a decision?
The answer is the timer tick. The kernel programs a hardware timer to interrupt the CPU at a fixed frequency, set by the build-time constant HZ, configurable as 100, 250, 300, or 1000 interrupts per second, with 250 as the default choice in the kernel's own configuration (kernel/Kconfig.hz). Every tick, the hardware forcibly transfers control to the kernel's timer interrupt handler no matter what the running task was doing. That handler updates time accounting, charges the current task for the CPU time it just consumed, and asks the scheduler: should someone else be running?
If the answer is yes, the kernel does not switch immediately inside the interrupt handler. It sets a flag on the current task, "reschedule needed" (TIF_NEED_RESCHED on x86), and returns from the interrupt. The actual switch happens at the next preemption point: on the way back from the interrupt to user mode, or at defined safe points inside the kernel. This two-step design keeps the switch machinery out of interrupt context, where the rules about what you may do are strict.
Two vocabulary terms fall out of this:
- A voluntary context switch happens when a task gives up the CPU on its own. It called
read()on an empty pipe, orsleep(), and blocked. The task asked to wait. - An involuntary context switch happens when the kernel takes the CPU away: the tick decided the task's time was up, or a more deserving task woke up. This taking-away is preemption.
Linux counts both per task; you will read the counters in the lab (/proc/PID/status, fields voluntary_ctxt_switches and nonvoluntary_ctxt_switches). One refinement worth knowing: on an idle CPU, a fixed tick is pure waste, so modern kernels suppress it when nothing is runnable ("tickless idle", NO_HZ). The tick is a tool the scheduler uses, not a metronome it worships.
What a context switch saves and restores [fundamental]§
A context switch is the act of suspending one task mid-thought and resuming another exactly where it left off, so precisely that neither can tell it happened. Chapter 8 introduced the task's kernel-side identity (task_struct); here is what physically moves.
The switch has two halves, and the split matters:
user task A running
│
▼ timer interrupt (or syscall that blocks)
┌─────────────────────────────────────────────────┐
│ HALF 1: kernel entry (already happened) │
│ CPU + entry code push A's user registers │
│ onto A's kernel stack (struct pt_regs) │
└─────────────────────────────────────────────────┘
│
▼ __schedule() picks task B
┌─────────────────────────────────────────────────┐
│ HALF 2: the switch proper (context_switch()) │
│ 1. switch address space: load B's page-table │
│ root (CR3 write on x86) if B's mm differs │
│ 2. switch_to(): swap kernel stack pointer and │
│ callee-saved registers A→B │
│ 3. __switch_to(): FPU/vector state, FS/GS │
│ thread-local-storage bases, per-CPU │
│ "current task" pointer │
└─────────────────────────────────────────────────┘
│
▼ return path pops B's pt_regs
user task B running, exactly where it stopped
Half 1 is not scheduler code at all. It is the ordinary kernel entry path from Chapter 7. By the time the scheduler runs, the interrupted task's user-visible registers are already parked on its kernel stack. Half 2 only needs to swap what the entry path did not: the kernel stack pointer itself, the callee-saved registers (more on why only those in the bridge notes), the floating-point/vector register state, the address-space root, and the thread-local-storage base registers.
What does it cost? Two different bills. The direct cost, executing the switch code and the CR3 write, is small: on modern x86_64 it is commonly measured in the low single-digit microseconds, though the exact figure varies enough with hardware and mitigations that you should measure your own machine rather than trust any published number. The indirect cost is the bigger and sneakier bill: the incoming task finds the CPU caches and TLB full of the previous task's data, and pays for the misses over its next thousands of instructions. (Address-space tagging, PCID on x86, lets the TLB keep entries from multiple tasks, softening the blow.) This indirect cost is why schedulers care about cache affinity: rerunning a task where it ran before is cheaper than moving it.
Hold onto the shape of that second bill. It is charged to the incoming task, it is spread over the thousands of instructions after the switch rather than at the switch, and no counter labels it "context switch." That is why a workload can look fine in a switch-count metric and still be slow, and it is the argument behind every "pin the thread and stop moving it" rule you will meet.
A short history in three schedulers [working]§
Linux's scheduler history compresses to three acts, and the direction of travel is the point.
Act one: O(1), 2002. Ingo Molnar's O(1) scheduler entered the 2.5 development series in early 2002 and shipped with 2.6.0. Its headline was constant-time decisions: 140 priority queues and a bitmap; find the highest non-empty queue, take the head. Fast. But which queue a task deserved was decided by interactivity heuristics: sleep a lot and you were "interactive" and got boosted. The heuristics were gameable and wrong at the margins, and tuning them became whack-a-mole.
Act two: CFS, 2007. Molnar's Completely Fair Scheduler was merged for kernel 2.6.23, released in October 2007. It threw the heuristic pile away and replaced it with an accounting identity: fairness computed, not guessed. It ran the world's Linux machines for sixteen years.
Act three: EEVDF, 2023. Peter Zijlstra proposed replacing CFS's core with EEVDF in early 2023, and the code was merged for kernel 6.6, released 29 October 2023. EEVDF stands for "Earliest Eligible Virtual Deadline First," and comes from a 1995 paper by Ion Stoica and Hussein Abdel-Wahab. The conversion was completed and refined in 6.12 (released 17 November 2024), which is the tree Coconut OS forks. 6.12 also merged two adjacent pieces worth a footnote: sched_ext, which lets a custom scheduler be loaded as BPF programs from userspace, and SCHED_DEADLINE "servers" to keep real-time tasks from starving normal ones.
So: what did CFS compute, and what did EEVDF fix?
CFS: fairness as arithmetic [working]§
CFS starts from a thought experiment: imagine an ideal CPU that could run all runnable tasks simultaneously, each at an equal fraction of its speed, so 4 tasks each run at 25% speed, continuously. No real CPU can do that; a real CPU runs one task at a time. CFS's move is to measure how far each task has fallen behind the ideal, and always run the task that is furthest behind.
The measuring stick is virtual runtime (vruntime): a per-task counter that advances while the task runs. For a normal-priority task it advances at wall-clock speed; for a high-priority task it advances slower (so the task must run longer to accumulate the same vruntime, i.e. it earns more CPU), and for a low-priority task faster. Priority becomes a weight on a clock, not a queue position. The task with the smallest vruntime is, by definition, the one the ideal CPU has shortchanged the most, so that is who runs next.
To find "smallest vruntime" quickly, CFS kept the runnable tasks in a red-black tree ordered by vruntime. That is a self-balancing binary search tree, the same species your data-structures course covered. The leftmost node is always the minimum, so picking the next task is "walk to the leftmost node" (cached, so it is O(1) in practice), and inserting a preempted task back costs O(log n). The kernel's own CFS design document describes the entire core this way: a time-ordered rbtree, pick the leftmost, repeat.
Sleeping tasks then need no special-case boost. While you sleep, your vruntime stands still while the runners' vruntimes advance, so when you wake, you are naturally on the far left of the tree and run promptly. Interactivity falls out of the arithmetic.
Mostly. Fairness over the long run says nothing about when within a fair share you run, which is the latency goal from the triangle. CFS had to bolt latency back on with wakeup-preemption heuristics and tunables, and a proposed "latency-nice" knob to let latency-sensitive tasks jump the queue. That growing pile of exceptions is exactly what EEVDF was brought in to delete.
EEVDF: latency without heuristics [working]§
EEVDF keeps CFS's virtual-time bookkeeping and adds two ideas that make latency a first-class output of the algorithm instead of a patch on top.
Idea one: lag, and eligibility. For each runnable task, compute its lag: the CPU time it should have received under the ideal fair split, minus what it actually received. Positive lag means the machine owes the task time; negative lag means the task has already been overserved. The rule: only tasks with zero or positive lag are eligible to run. An overserved task is not punished forever. It waits until the ideal clock catches up with what it already consumed. This one rule replaces CFS's fairness maintenance with a hard invariant: you cannot get ahead and stay ahead.
Idea two: virtual deadlines. Among eligible tasks, which one runs? Each task has a time slice, a per-task request for how much CPU it wants in one go, which can be small for latency-sensitive tasks and large for batch work. EEVDF computes each task's virtual deadline: the point on the virtual clock by which its slice ought to complete (its eligible time plus its slice, weighted). Then it runs the eligible task with the earliest virtual deadline.
Watch what the slice length now does. A task that asks for a short slice gets a near deadline, so it runs soon. But a short slice means it is descheduled quickly and, having been served, becomes ineligible until fairness catches up. It gets low latency without getting extra throughput. A batch task with a long slice runs later but longer, with fewer switches. The fairness/latency trade-off has moved from scheduler heuristics into a per-task parameter with defined semantics, precisely the thing CFS's latency-nice patches were groping toward, delivered "in a clean, algorithmic way," as the kernel's EEVDF changelog put it.
That trade is the sentence to keep. A latency-sensitive task buys a shorter wait by asking for less CPU per turn, not by asking for more CPU. Anyone who has watched a "make it faster" change consist of raising a thread's priority and then measured no improvement has met the other side of the same equation.
One implementation note that will matter when you read the source: the red-black tree survives, but in the 6.12 tree it is ordered by virtual deadline rather than vruntime, with an augmented per-subtree minimum vruntime so the picker can skip ineligible subtrees while descending. "Pick leftmost" became "pick earliest eligible deadline."
The policy zoo [working]§
Everything above describes the fair class, the default. Linux actually runs several scheduling classes in strict precedence: stop (kernel-internal) > deadline > real-time > fair > idle. If any task of a higher class is runnable, no task of a lower class runs at all. Within the classes, userspace-visible policies (set with sched_setscheduler(2)/sched_setattr(2), or the chrt tool):
| Policy | Class | Selector knob | Semantics | Typical use |
|---|---|---|---|---|
SCHED_OTHER | fair (EEVDF) | nice −20…+19 | Weighted fair share; default for everything | Nearly all processes |
SCHED_BATCH | fair | nice −20…+19 | Fair share, but assumed CPU-bound; small wakeup penalty, fewer preemptions | Compile farms, batch jobs |
SCHED_IDLE | fair | none | Runs only in otherwise-idle time; below even nice +19 | Background indexing, scavengers |
SCHED_FIFO | real-time | priority 1-99 | Runs until it blocks, yields, or a higher priority preempts; no time slicing | Hard latency work (audio, control loops) |
SCHED_RR | real-time | priority 1-99 | FIFO plus round-robin time quantum among equal-priority peers | RT tasks that must share a level |
SCHED_DEADLINE | deadline | runtime/deadline/period | Earliest-deadline-first with a bandwidth guarantee (GEDF + CBS); admission-controlled; preempts all of the above | Periodic work with hard timing (since Linux 3.14) |
Three footnotes the table cannot hold. First, nice is the classic Unix priority dial: −20 is greediest, +19 is meekest, and under the fair class it maps to the vruntime weight, which is a suggestion about shares and not a guarantee. Second, real-time policies are dangerous by construction: a SCHED_FIFO task in a spin loop outranks every normal task forever. As a seatbelt, the kernel reserves about 5% of each second for the normal classes, so a runaway RT task leaves you a sliver of machine to recover with. The classic mechanism was RT throttling: the RT class could burn only 950,000 µs of each 1,000,000 µs window (/proc/sys/kernel/sched_rt_runtime_us). In 6.12 that job changed hands. A per-CPU deadline-class "fair server" with an equivalent 50 ms-per-second reservation now serves the fair class (commit 5f6bd380c7bd removed default RT bandwidth control), and the old throttling path applies only under CONFIG_RT_GROUP_SCHED. The sysctl still exists and still reads 950000, but on a default 6.12 config it is no longer the mechanism enforcing the cap. You will see the 5% either way in the lab. Third, SCHED_RR's quantum is queryable via sched_rr_get_interval(2) and tunable via /proc/sys/kernel/sched_rr_timeslice_ms.
The "no time slicing" cell in the SCHED_FIFO row is the one that surprises people in production. Two FIFO threads at the same priority do not take turns. The first one to get the CPU keeps it until it blocks or yields, so a busy-poll loop under SCHED_FIFO will hold a core against its own equal-priority sibling indefinitely. That is SCHED_RR's reason for existing, and it is why lab 2 is worth running before you reach for chrt on anything real.
Group fairness is the last piece of policy. Per-task weights compose badly: a user who starts 100 processes should not get 100 shares against your 1. Control groups (cgroup v2, Chapter 18) fix this hierarchically: the cpu controller's cpu.weight file (range 1 to 10,000, default 100) assigns a fair-class weight to an entire group of tasks. The scheduler runs fairness between groups first, then within each group, recursively. Twice the weight, twice the CPU share for the whole subtree, no matter how many tasks are inside. Hold onto this idea; it is the conceptual parent of Coconut's SCHED_AGENT.
Many CPUs: runqueues, balancing, domains [advanced]§
Everything so far pretended one CPU. Real machines have dozens, and the naive design, one global queue of runnable tasks locked by whichever CPU is choosing, dies of lock contention exactly when the machine is busiest.
So Linux gives every CPU its own runqueue (struct rq): its own rbtree, its own lock, its own scheduling decisions, no cross-CPU coordination on the hot path. The cost of that independence is drift: one CPU's queue can pile up while another idles. Restoring balance is the load balancer's job: periodically (and at moments like a CPU going idle), a CPU examines others' queues and pulls tasks over.
But pulling a task is not free. Remember the indirect context-switch cost, because a migrated task abandons its warm caches. So the balancer is deliberately non-uniform, guided by a hierarchy of scheduling domains that mirrors the hardware's cost structure:
NUMA domain (separate memory controllers - migrate reluctantly)
└─ package domain (same chip, shared last-level cache)
└─ cluster/LLC (cores sharing L2/L3 - migrate cheaply)
└─ SMT domain (hyperthread siblings - migrate freely)
Balancing is aggressive at the bottom (moving between hyperthread siblings barely costs anything) and reluctant at the top. The top level is NUMA, Non-Uniform Memory Access: machines where each socket has its own attached RAM and touching the other socket's RAM is markedly slower. Migrating a task across a NUMA boundary strands its memory on the far node, so the scheduler and the memory subsystem negotiate (Chapter 13 picks this thread up). For now the takeaway: on big machines, where a task runs is a memory-placement decision wearing a scheduling costume.
This subsection is marked advanced because you can build a correct mental model of the scheduler without it, and nothing in the labs depends on it. Skip it if you are working on a laptop and come back the first time you are handed a two-socket box.
The real thing in Linux [working]§
Everything in this chapter lives under kernel/sched/ in the 6.12 tree. A map, with the names you can grep for (paths and symbols verified against v6.12 source):
| Where | What |
|---|---|
kernel/sched/core.c | The class-independent engine. __schedule() is the heart: every context switch on the machine funnels through it. context_switch() performs half 2 of the switch and calls the arch-specific switch_to(). |
kernel/sched/fair.c | The fair class, EEVDF since 6.6. pick_eevdf() walks the deadline-ordered tree for the earliest eligible entity; entity_eligible() / vruntime_eligible() implement the lag rule; update_deadline() computes virtual deadlines. The load balancer also lives here: sched_balance_rq(), sched_balance_domains(), sched_balance_newidle(). |
kernel/sched/rt.c, deadline.c, idle.c, stop_task.c | The other scheduling classes, one file each. |
kernel/sched/ext.c | sched_ext, new in 6.12: BPF-programmable scheduling. |
kernel/sched/sched.h | struct rq (the per-CPU runqueue) and struct cfs_rq, whose tasks_timeline is the rb_root_cached red-black tree. |
include/linux/sched.h | struct sched_entity, embedded in every task_struct: vruntime, vlag (the lag), deadline, slice, and run_node (the task's rbtree linkage). The EEVDF vocabulary is literally the field list. |
arch/x86/kernel/process_64.c | __switch_to(), the x86_64 tail of the switch: FPU state, FS/GS bases, per-CPU current pointer. |
Two source-reading notes. First, a task's class is a pointer to a struct sched_class of function pointers (pick_next_task, enqueue_task, task_tick), so core.c never knows which algorithm it is driving; the policy zoo is a vtable. (Your compiler coursework will recognize the dispatch pattern immediately.) Second, the historical naming shows through: the fair runqueue is still cfs_rq and the file is still fair.c, even though the algorithm inside has been EEVDF since 6.6. Kernel code renames reluctantly; read structure names as fossils, not documentation.
Two names are enough to start reading: pick_eevdf() in fair.c for "who runs next," and __schedule() in core.c for "how the switch happens." Open pick_eevdf() first. It is short, and the lag and deadline vocabulary you just learned is the vocabulary in the code, which is a rarer alignment in this tree than you would hope.
Coconut tie-in [working]§
Coconut OS's scheduling story is a planned scheduling class, SCHED_AGENT, still in spec phase (04-HLD and 05-LLD) with nothing implemented yet.
The lineage is kvwarden, Coconut Labs' CUDA inference broker, whose central problem was tenant fairness: many principals submitting GPU inference work, none of whom may starve another, with priority expressed as an enforceable share rather than an honor-system knob. That is precisely the cgroup cpu.weight shape: fairness between principals first, between tasks second. Coconut's design bet is that for a machine whose workloads are AI agents, the principal should be a kernel primitive, not a control-group convention layered on top.
Concretely, per the spec: an agent created via agent_spawn (syscall 472, the wired entry in the 472-479 family from Chapter 7) is a kernel object under kernel/agent/, and SCHED_AGENT would make that object the unit of CPU fairness. Every task belonging to an agent draws from the agent's share; the agent's share derives from its attested identity and capability grant, not from a nice value any of its processes chose for itself. Fairness by construction, not by convention: an agent that spawns a thousand workers gains nothing on an agent running one, and a misbehaving agent's blast radius is capped at its own share, the same invariant flavor as EEVDF's lag rule applied one level up. The planned substrate is the 6.12 fair-class entity machinery this chapter just unpacked, which is one reason the fork tracks 6.12: the EEVDF conversion is complete there, and group-entity scheduling is mature. Whether SCHED_AGENT lands as a true new class beside fair.c or as an agent-keyed weighting layer over it is an open LLD question. The invariant, agents as schedulable principals, is the locked part.
Lab [working]§
Host is macOS, and everything here reads /proc, so work inside your Chapter 0 lab bench: the QEMU Linux guest (preferred, because labs 2 and 3 need root and real scheduling control) or, for lab 1 only, a Linux container (docker run --rm -it --cpus=2 debian bash). Predict before you run; that is the whole method.
Lab 1: more spinners than CPUs [working]§
Predict first: with 8 CPU-burn tasks on 2 CPUs, what does the 1-minute load average converge toward, and what share of a CPU does each task get? (Linux's load average counts runnable and uninterruptible-sleep tasks, averaged with exponential decay over 1/5/15 minutes.)
nproc # note M, your CPU count (assume 2 below)
for i in $(seq 1 8); do sh -c 'while :; do :; done' & done
sleep 60
cat /proc/loadavg
Expected:
8.02 5.71 2.60 9/143 1730
The 1-minute figure climbs toward ~8 (8 always-runnable tasks); the 5- and 15-minute figures lag behind because they decay more slowly. Each spinner gets roughly 2 CPUs / 8 tasks = 25% of a CPU. Check with top. Now read one spinner's scheduler ledger and its context-switch counters:
PID=$(jobs -p | head -1)
cat /proc/$PID/schedstat; sleep 5; cat /proc/$PID/schedstat
grep ctxt /proc/$PID/status
Expected:
4816321047 14273390122 1502
4941295310 14648012877 1541
voluntary_ctxt_switches: 2
nonvoluntary_ctxt_switches: 1539
The three schedstat fields are cumulative nanoseconds on-CPU, nanoseconds waiting on a runqueue, and timeslices run. Prediction to check: with 4× oversubscription, field 2 (waiting) should grow about 3× faster than field 1 (running), because each task runs 1 unit then waits behind 3 peers. And the context switches are almost all nonvoluntary: a spin loop never blocks, so every switch is the tick taking the CPU away. Kill the spinners (kill $(jobs -p)) before lab 2.
Those two schedstat fields are worth remembering as a pair, because their ratio is a saturation measurement you can take on any process on any Linux box without installing anything. Field 2 climbing faster than field 1 means the task is spending more time queued than running, which is the machine telling you it is oversubscribed.
Lab 2: SCHED_FIFO starves the fair class, almost [working]§
Predict first: pin one normal spinner and one SCHED_FIFO spinner to the same CPU. Does the normal one get 0% CPU, or something else? Recall the 95% seatbelt.
sysctl kernel.sched_rt_runtime_us # confirm the default seatbelt
taskset -c 0 sh -c 'while :; do :; done' & # SCHED_OTHER victim, CPU 0
OTHER=$!
sudo taskset -c 0 chrt -f 50 timeout 30 sh -c 'while :; do :; done' &
top -b -n 3 | grep -E "sh$|COMMAND" | head -6
Expected:
kernel.sched_rt_runtime_us = 950000
PID USER PR NI ... %CPU COMMAND
1799 root -51 0 ... 95.0 sh
1797 user 20 0 ... 5.0 sh
Not 0%. 5%. The FIFO task (priority shown as −51, i.e. RT 50) outranks the fair class absolutely, but the victim scavenges a reserved slice: on 6.12 the per-CPU deadline-class fair server grants the normal classes 50 ms of every second. (Pre-6.12 kernels produced the same 5% via RT throttling at 950 ms per 1000 ms; the sysctl survives but no longer bites on default configs.) That reservation is a kernel default saving you from yourself. The timeout 30 is the second seatbelt. One trap worth knowing: the classic trick of sudo sysctl -w kernel.sched_rt_runtime_us=-1 no longer produces true starvation on 6.12. The fair server keeps serving its 50 ms regardless, so the victim's top line stays near 5.0, not 0.0. To watch full starvation (QEMU guest only, never a machine you care about), zero the fair server itself: echo 0 | sudo tee /sys/kernel/debug/sched/fair_server/cpu0/runtime, repeated per CPU. Then the victim pins at 0.0 and even your shell freezes until timeout fires. Restore each runtime to its default 50000000 (ns) afterwards.
The 5% is the number to carry out of this lab. It means an SCHED_FIFO thread never gets a whole core no matter what you do to the sysctl, and it means the machine stays reachable when someone ships a busy-poll loop at RT priority. Both halves of that matter, and which half you notice first depends on whether you are the one who shipped the loop.
Lab 3: watch latency directly with perf sched [working]§
Install perf in the guest (Debian: apt install linux-perf). Run a mixed workload, spinners as antagonists plus a task that wakes frequently, and record scheduler events:
for i in $(seq 1 4); do sh -c 'while :; do :; done' & done
sudo perf sched record -- sh -c 'for i in $(seq 1 200); do sleep 0.01; done'
sudo perf sched latency | head -8
kill $(jobs -p)
Expected:
-----------------------------------------------------------------------------------------
Task | Runtime ms | Count | Avg delay ms | Max delay ms |
-----------------------------------------------------------------------------------------
sh:1832 | 142.3 ms | 412 | avg: 1.842 ms| max: 12.211 ms |
sleep:1840 | 1.1 ms | 4 | avg: 0.310 ms| max: 0.702 ms |
"Delay" is exactly the quantity EEVDF exists to control: the gap between became runnable and actually ran. Predict first: which shows the larger max delay, the CPU-hog sh processes or the constantly-waking sleeps? Then check whether EEVDF's wakeup handling matches your prediction. The frequently-sleeping tasks should show markedly smaller delays, because a waker with unspent lag is eligible immediately and its short remaining slice yields a near deadline. perf sched timehist (same recording) shows every individual switch if you want the microscope.
You can now measure the thing people usually argue about. Next time someone says a service is slow because "the scheduler isn't giving it enough CPU," you have a way to check: perf sched latency separates how much a task ran from how long it waited to run, and those two numbers point at different fixes. Run it once against something real of your own with the four spinners going and once without them. The delta between those two max-delay columns is what "a busy machine" costs you, in milliseconds, on your hardware, which is a better number than any figure printed in a book.
Bridge notes [fundamental]§
| You already know | What is new at this scale |
|---|---|
| SysTick interrupt driving a FreeRTOS-style tick on a Cortex-M | Same mechanism, since the tick is only a periodic interrupt, but the handler feeds a fairness algorithm, not a fixed-priority table. And on servers the tick itself is now conditional (tickless idle). |
| RTOS fixed-priority preemptive scheduling, rate-monotonic analysis | That entire worldview survives intact as SCHED_FIFO/SCHED_RR. It stopped being the whole scheduler and became one class, outranked by SCHED_DEADLINE (which is EDF from your real-time coursework, plus an admission test and bandwidth caps). |
| EDF (earliest deadline first) from real-time theory | EEVDF is EDF transplanted into virtual time: deadlines computed on a weighted fairness clock rather than the wall clock. Same "earliest deadline wins" spine, new definition of deadline. |
| Calling conventions: caller-saved vs callee-saved registers, from compiler codegen | Load-bearing in the kernel: switch_to() runs at a known function-call boundary, so the compiler has already spilled the caller-saved registers, and the switch code needs to save only the callee-saved set plus the stack pointer. The context switch is cheap partly because the calling convention did half the work. |
| ISR prologue/epilogue saving registers on an MCU | Half 1 of the context switch is exactly that (pt_regs). The new part is half 2: swapping kernel stacks, FPU state, TLS bases, and the address space. An MCU with no MMU never had an address space to swap. |
| Vtables / dispatch tables from compiler construction | struct sched_class is a hand-rolled vtable; core.c is written against the interface and never names an algorithm. That is how five policies coexist in one kernel. |
| One core, one ready queue | Per-CPU runqueues plus a cost-aware balancer over a domain hierarchy (SMT → cache → package → NUMA). Scheduling becomes a data-locality problem; the queue is the easy part. |
Sources [fundamental]§
- https://lwn.net/Articles/241085/: LWN, "CFS scheduler merged" (July 2007): CFS merged for 2.6.23, replacing the O(1) scheduler's interactivity heuristics.
- https://kernelnewbies.org/Linux_2_6_23: 2.6.23 release (October 2007) shipping CFS; SCHED_IDLE addition.
- https://lwn.net/Articles/925371/: LWN, "An EEVDF CPU scheduler for Linux" (March 2023): Zijlstra's proposal; lag, eligibility, virtual deadlines; the 1995 Stoica & Abdel-Wahab paper; replacement of latency-nice heuristics.
- https://kernelnewbies.org/Linux_6.6: Linux 6.6 released 29 October 2023 with EEVDF merged.
- https://www.phoronix.com/news/Linux-6.6-EEVDF-Merged: EEVDF merged for 6.6; "clean, algorithmic way" vs CFS heuristics framing.
- https://kernelnewbies.org/Linux_6.12: Linux 6.12 released 17 November 2024: EEVDF completion, sched_ext merge, SCHED_DEADLINE servers.
- https://docs.kernel.org/scheduler/sched-design-CFS.html: kernel CFS design doc: ideal multitasking CPU model, vruntime-ordered rbtree, pick leftmost.
- https://docs.kernel.org/scheduler/sched-eevdf.html: kernel EEVDF doc: relationship to CFS, per-task slice semantics.
- https://man7.org/linux/man-pages/man7/sched.7.html: sched(7): policy list, nice range −20…+19, RT priorities 1-99, SCHED_DEADLINE since 3.14 (GEDF+CBS), SCHED_BATCH/SCHED_IDLE semantics, sched_rt_runtime_us.
- https://docs.kernel.org/scheduler/sched-rt-group.html: RT throttling defaults: 950,000 µs runtime per 1,000,000 µs period, 5% reserved for non-RT (the pre-6.12 default mechanism).
- https://github.com/torvalds/linux/commit/5f6bd380c7bdbe10f7b4e8ddcceed60ce0714c6d: sched/rt: default RT bandwidth control removed in 6.12; the deadline-class fair server takes over the reservation.
- https://docs.kernel.org/admin-guide/cgroup-v2.html: cgroup v2 weight semantics: range [1, 10000], default 100.
- https://docs.kernel.org/scheduler/sched-stats.html: /proc/PID/schedstat's three fields (on-CPU ns, runqueue-wait ns, timeslice count).
- https://man7.org/linux/man-pages/man1/perf-sched.1.html: perf sched record/latency/timehist semantics.
- https://elixir.bootlin.com/linux/v6.12/source/include/linux/sched.h: struct sched_entity fields in v6.12: vruntime, vlag, deadline, slice, run_node (verified against the v6.12 tag).
- https://elixir.bootlin.com/linux/v6.12/source/kernel/sched/fair.c: pick_eevdf(), entity_eligible(), update_deadline(), sched_balance_rq(), sched_balance_domains() in v6.12; deadline-ordered tree with augmented min_vruntime.
- https://elixir.bootlin.com/linux/v6.12/source/kernel/sched/core.c: __schedule(), context_switch(), switch_to() call in v6.12.
- https://elixir.bootlin.com/linux/v6.12/source/kernel/sched/sched.h: struct rq, struct cfs_rq, tasks_timeline as rb_root_cached in v6.12.
- https://elixir.bootlin.com/linux/v6.12/source/arch/x86/kernel/process_64.c: __switch_to() on x86_64.
- https://elixir.bootlin.com/linux/v6.12/source/kernel/Kconfig.hz: HZ choices 100/250/300/1000, default 250.
- https://lkml.iu.edu/hypermail/linux/kernel/0201.1/2288.html: Molnar's O(1) scheduler patch posting, January 2002.