Chapter F.3: Memory from First Principles§

Authors: Shrey Patel and Jay Patel, Coconut Labs
Book: Think Like an OS (see book/00-INDEX.md)

This chapter closes the single biggest mental-model gap between application programming and systems programming: what memory actually is. You will leave knowing that memory is one huge array of bytes with numeric addresses, that a pointer is just such a number, which order the bytes of an integer land in (and why your machine's answer is "backwards"), what the stack and the heap physically are, why a struct with a char and a long occupies 16 bytes instead of 9, and how to recognize the two memory bugs (buffer overflow and use-after-free) that account for a shocking fraction of all security advisories. Everything is grounded in runnable C you can execute in a Docker container in under ten minutes.

The problem§

If you come from web, data, or mobile work, your runtime has been lying to you, politely and for your own good. In Python, JavaScript, Java, or Swift, a "variable" is a name bound to an object, objects live "somewhere," and a garbage collector cleans up. You never see an address. You never decide how big anything is or where it goes.

An operating system kernel gets none of that comfort. The kernel is the thing that implements the comfortable illusion, so it must work at the layer below it: raw bytes at numeric addresses. Almost every later chapter of this book stands on that layer. Chapter 3's virtual-memory discussion is meaningless if you don't first believe that an address is a number. Chapter 11's page tables and VMAs are transformations of addresses, and you need to know what is being transformed. Kernel code is full of structs whose exact byte layout is a contract between components (and, for syscall arguments, between the kernel and every userspace program ever compiled); if padding and alignment are mysteries, those contracts are unreadable. And the two classic memory bugs are not trivia: they are the daily bread of kernel security work.

Three members in a struct occupy 24 bytes or 16 depending only on the order you declared them. Stack allocation costs one arithmetic instruction, while malloc is a library routine that sometimes enters the kernel, and free usually does not hand memory back to the OS at all. Those are not kernel trivia; they sit on the hot path of anything written for low latency. For kernel work the same material is the floor under Chapter 3 and Chapter 11, with no route around it.

So this chapter unfolds memory slowly, from "it's an array" to "here is my process's actual memory map," with no step skipped.

Unfolded§

Memory is one huge array of bytes§

Physically, your machine's RAM is billions of tiny cells, each holding one byte: 8 bits, a value from 0 to 255. Conceptually, ignore the physics and picture a single enormous array:

index:    0        1        2        3        4        5    ...
        ┌────────┬────────┬────────┬────────┬────────┬─────
memory: │  0x48  │  0x69  │  0x21  │  0x00  │  0x7f  │ ...
        └────────┴────────┴────────┴────────┴────────┴─────

The index of a cell is called its address. That's the entire concept. An address is not a special kind of value; it is an integer that happens to be used as an index into this array. When people say "the string is at address 0x5586a3f2b004," they mean exactly what "the element at index 4" means for a Python list, with a much bigger array and the index written in hexadecimal, because addresses are large and hex lines up neatly with bytes (two hex digits per byte).

On a 64-bit machine, addresses are 64-bit numbers, which is why a pointer occupies 8 bytes there.

A pointer is an address is a number§

C exposes addresses directly, and this is where newcomers expect mystique. There is none. A pointer is a variable whose value is an address, and an address is a number. The & operator means "give me the address of this variable." The * operator means "go to this address and use what's there."

c
#include <stdio.h>

int main(void) {
    int x = 42;
    int *p = &x;                                  /* p holds x's address */

    printf("value of x:            %d\n", x);
    printf("address of x (&x):     %p\n", (void *)&x);
    printf("value of p:            %p\n", (void *)p);
    printf("p as a plain integer:  %lu\n", (unsigned long)p);
    printf("value at p (*p):       %d\n", *p);
    return 0;
}

Expected: (your addresses WILL differ, and there is more on why in the virtual-memory teaser)

value of x:            42
address of x (&x):     0x7ffd8c1a4a5c
value of p:            0x7ffd8c1a4a5c
p as a plain integer:  140726953986652
value at p (*p):       42

Look at the last two address lines together. 0x7ffd8c1a4a5c and 140726953986652 are the same number in two notations. That is the whole demystification: a pointer prints as hex by convention, but you can cast it to an integer and do arithmetic on it, because underneath it is one. Everything else about pointers (arithmetic, arrays, dereferencing) falls out of "it's an index into the big byte array."

Endianness: which end of a number comes first§

An int is 4 bytes, but memory is addressed one byte at a time. So storing the value 0x12345678 raises a question with no obviously right answer: which of its four bytes (12, 34, 56, 78) goes at the lowest address?

ConventionByte at lowest addressMemory order for 0x12345678Used by
Little-endianLeast-significant byte (78)78 56 34 12x86_64; ARM64 in practice
Big-endianMost-significant byte (12)12 34 56 78Network byte order; SPARC; older POWER

The names come from a 1980 note by Danny Cohen, "On Holy Wars and a Plea for Peace" (IEN 137), which borrowed the warring Big-Endians and Little-Endians (factions who cracked their eggs at different ends) from Swift's Gulliver's Travels. The joke stuck because the choice genuinely is arbitrary; what matters is that both sides of any byte exchange agree.

The two machines you will actually touch: x86_64 is little-endian, full stop. The System V AMD64 ABI specifies it. ARM64 (AArch64) can be configured either way for data, but little-endian is the default and effectively universal: Linux distributions for ARM64 ship little-endian. So on both a cloud x86 box and an Apple Silicon Mac, 0x12345678 sits in memory as 78 56 34 12. Verify it yourself:

c
#include <stdio.h>

int main(void) {
    unsigned int x = 0x12345678;
    unsigned char *p = (unsigned char *)&x;   /* view the same 4 bytes byte-by-byte */
    for (int i = 0; i < 4; i++)
        printf("byte at address+%d: %02x\n", i, p[i]);
    return 0;
}

Expected on x86_64 or ARM64:

byte at address+0: 78
byte at address+1: 56
byte at address+2: 34
byte at address+3: 12

The value looks "reversed" in a hex dump. It isn't reversed; it's stored little end first. Once you can read 78 56 34 12 and think "little-endian 0x12345678" without pausing, hex dumps of packets, file formats, and kernel structures stop being noise.

(Viewing any object through an unsigned char *, as above, is explicitly legal C; character types are the standard's escape hatch for inspecting raw bytes.)

The classic process memory layout: the cartoon§

Every running program (a process) gets an address space that is conventionally drawn like this:

high addresses (e.g. 0x7ffc........)
┌───────────────────────────────┐
│  stack        (grows DOWN ↓)  │  function locals, call frames
├───────────────────────────────┤
│         (unused gap)          │
├───────────────────────────────┤
│  shared libraries / mmap area │  libc.so, the dynamic linker,
│                               │  large allocations
├───────────────────────────────┤
│         (unused gap)          │
├───────────────────────────────┤
│  heap         (grows UP ↑)    │  malloc() lives here
├───────────────────────────────┤
│  bss          (zeroed data)   │  globals with no initializer
├───────────────────────────────┤
│  data         (init'd data)   │  globals with initializers
├───────────────────────────────┤
│  text                         │  your machine code, read+execute
└───────────────────────────────┘
low addresses

Reading bottom-up:

An honest disclaimer before you fall in love with this picture: it is a cartoon. It teaches the right vocabulary and roughly the right ordering, and the lab below will show your own process matching it. But the real kernel does not maintain "a heap segment" and "a stack segment" as first-class objects; it maintains a list of VMAs (virtual memory areas), dozens of them, one per mapping, and it deliberately randomizes where the regions land at every launch (ASLR, or address-space layout randomization) so attackers can't hardcode addresses. Chapter 11 replaces this cartoon with the real machinery. Until then, the cartoon is a fine mental scaffold.

The stack: what "on the stack" physically means§

When a function is called, it needs scratch space: its local variables, plus bookkeeping like where to resume when it returns. That space is called its stack frame, and it is allocated by the cheapest mechanism imaginable: the CPU keeps a register (the stack pointer) holding the address of the current top of the stack, and "allocating" a frame is just subtracting from that register. Sixteen bytes of locals? Subtract 16. Returning from the function? Add it back. One arithmetic instruction each way. That is why stack allocation is nearly free, and why locals in C have no malloc/free ceremony.

Two consequences matter:

  1. Lifetime is the function call. A local variable's bytes are reclaimed the instant the function returns: the stack pointer moves back past them, and the very next call will scribble its own frame over them.
  2. Therefore, returning a pointer to a local is a bug. Always. The pointer you return holds the address of memory that no longer belongs to that variable:
c
int *broken(void) {
    int x = 42;
    return &x;        /* x's storage evaporates at the closing brace */
}

The C standard is precise about this: a pointer's value becomes indeterminate the moment the object it points to reaches the end of its lifetime. The cruel part is that the bug often appears to work, because the stale bytes may survive until the next function call overwrites them. That is exactly why you must recognize the pattern by sight rather than by testing.

The heap: what malloc actually does§

When you need memory whose lifetime is not tied to a function call, because it must outlive its creator or its size isn't known until runtime, you use the heap. malloc(n) returns the address of a block of at least n bytes that is yours until you call free on it.

malloc is not a system call. It is a library function (in glibc on Linux) that manages a big region on your behalf, carving it into chunks. Under the hood, per the Linux man pages: small and medium requests are carved from the heap region, extended when needed via the sbrk/brk system call, which pushes the heap's top boundary upward. Allocations larger than a threshold (M_MMAP_THRESHOLD, 128 KiB by default) are instead served as private anonymous mappings via mmap, off in the shared-library area of the cartoon. Both of those paths can enter the kernel, and the mmap one always does, which is one reason code that cares about its worst-case timing allocates its buffers up front and reuses them.

free(p) is widely misunderstood, so let's state precisely what it does and does not do:

free(p) doesfree(p) does not
Mark the chunk as available for reuse by a later mallocZero or otherwise erase your data (the bytes linger)
Let the allocator thread the chunk into its free listsImmediately return memory to the OS (usually)
Possibly trim the heap top via sbrk, but only when the contiguous free space at the top exceeds M_TRIM_THRESHOLD (128 KiB by default)Make your pointer null or invalid-looking (p still holds the old address)
Immediately munmap the chunk if it was a big mmap-served oneCheck that p was valid (double-free is undefined behavior)

Two of those non-actions have teeth. Because freed bytes are not erased, secrets (keys, passwords) linger in memory after free. Worse, the allocator repurposes the freed chunk's user-data area for its own bookkeeping, storing free-list link pointers right where your data used to be. Because your pointer variable still holds the old address, nothing stops you from using it, which brings us to use-after-free below.

Arrays, strings, and structs in memory§

Arrays are the honest ones: int a[4] is exactly 16 contiguous bytes, four 4-byte ints, no header, no length field, nothing. a[i] compiles to "start address + i × element size." This is also why C arrays don't know their own length: there is nowhere for the length to live.

Strings in C are just byte arrays with a convention. The C standard's definition: a string is a contiguous sequence of characters terminated by (and including) the first null character: a zero byte, written '\0' and called NUL. "Hi!" occupies four bytes: 48 69 21 00. Every string function (strlen, strcpy, printf("%s")) works by scanning forward until it hits the zero byte. Lose the terminator and those functions happily run off the end of your buffer into whatever lies beyond. A whole class of bugs comes from one missing byte.

Structs are where newcomers get ambushed by alignment. Hardware and the platform ABI want an N-byte value to sit at an address divisible by N. The System V AMD64 ABI (the rulebook for x86_64 Linux) sets these:

TypeSizeRequired alignment
char11
short22
int44
long88
pointer88
double88

The struct layout algorithm, straight from the ABI: place each member at the next offset satisfying its alignment, inserting padding bytes as needed; the whole struct's alignment is the largest member alignment; and the struct's total size is rounded up to a multiple of that. Now the classic surprise stops being surprising:

c
struct demo {
    char c;      /* offset 0                          - 1 byte  */
                 /* offsets 1..7: PADDING             - 7 bytes */
    long l;      /* offset 8 (must be 8-aligned)      - 8 bytes */
};               /* sizeof(struct demo) == 16, not 9            */

The long needs an 8-aligned offset, so seven dead bytes are inserted after the char. Member order matters, and you control it:

c
struct wasteful  { char a; long l; char b; };  /* 24 bytes: 7 pad + 7 pad */
struct compact   { long l; char a; char b; };  /* 16 bytes: 6 pad at end  */

Same three members, 8 bytes difference. In kernel code, where a struct may exist in millions of instances or cross the user/kernel boundary as a binary contract, people order members deliberately and treat padding as part of the ABI. One more honesty note you'll want later: the standard does not guarantee padding bytes hold any particular value, or that they survive member assignment. Never let secrets or meaning hide in padding.

The two bugs you must recognize on sight§

Bug 1: buffer overflow. Writing past the end of an array.

c
#include <string.h>
void greet(const char *name) {
    char buf[8];
    strcpy(buf, name);   /* copies until NUL - no length check at all */
}
int main(void) { greet("far longer than eight bytes"); return 0; }

strcpy copies until it finds the NUL terminator, and buf holds 8 bytes; the rest lands in the adjacent stack memory. What gets corrupted: the neighboring locals, the saved frame pointer, and above all the saved return address, the stack slot holding where this function should jump back to. Overwrite that with attacker-chosen bytes and the function "returns" to code of the attacker's choosing. This single pattern is the ancestral security vulnerability. The fix is always the same shape: bounded operations (strlcpy, snprintf, explicit length checks).

Bug 2: use-after-free. Using memory through a pointer after freeing it.

c
#include <stdlib.h>
int main(void) {
    char *p = malloc(64);
    free(p);             /* chunk returned to the allocator...      */
    p[0] = 'X';          /* ...but p still holds the address. BOOM. */
    return 0;
}

Recall what free actually did: the bytes still exist, the allocator now stores its free-list pointers in them, and the next malloc may hand the same bytes to unrelated code. So this write corrupts either allocator metadata (crashing some later, unrelated malloc, which is miserable to debug) or another live object that reused the chunk (attacker heap-grooming territory). Like the returning-a-pointer-to-a-local bug, it frequently "works" in testing. Recognize the shape: any pointer that outlives a free of what it points to is a loaded gun. In kernel code this bug class is called UAF and it is a staple of exploit chains.

Both bug shapes are now something you can name on sight in a diff, with the mechanism attached. You can write that a pointer outlives the free of what it points to and that the allocator has already stored its free-list links in those bytes, which is a review comment that gets acted on.

Virtual memory: the teaser§

One puzzle remains. If memory is one big array, and your process uses address 0x7ffd8c1a4a5c... what stops another process from reading it?

Answer: every process gets its own private array. The addresses your program sees are virtual: a per-process fiction that the CPU and kernel translate to real physical RAM on every single access. Two processes can both use address 0x400000 and be touching completely different physical bytes; neither can even name the other's memory. This is also why your printed addresses change run to run (ASLR shuffles the fiction) and how the same physical copy of libc appears inside thousands of processes at once. How the translation works, through page tables, TLBs, and faults, is Chapter 3 and Chapter 11. For now, hold one sentence: every address in this chapter was virtual, and the kernel owns the mapping.

The real thing in Linux§

Everything above is inspectable on a stock Linux box with a Linux 6.12-era kernel:

Coconut tie-in§

Coconut's agent_* syscalls (472 to 479) pass uapi structs across the user/kernel boundary, where member order, padding, and alignment are the ABI. The struct-layout rules in this chapter are exactly what the 05-LLD struct definitions are written against. The audit pipeline's BLAKE3 hash chain runs over serialized record bytes, so byte order and layout discipline decide whether two implementations hash identically. And the v1 mm tier-aware hooks attach to precisely the VMA machinery you just previewed in /proc/self/maps.

Lab§

Runs in Docker Linux, recommended because Docker Desktop on macOS runs a Linux VM, so this works identically on a Mac. Labs 2 and 3 also run natively on macOS with clang; Lab 1 needs /proc, which macOS doesn't have (the rough macOS analog is vmmap <pid>). Type these rather than read them. The moments that teach are the ones where your output disagrees with the prediction you wrote down.

sh
mkdir -p /tmp/f3lab && cd /tmp/f3lab
docker run --rm -it -v "$PWD":/work -w /work gcc:14 bash

Lab 1: map your own process (red-then-green)§

Predict first: rank these five addresses lowest to highest: a function, an initialized global, an uninitialized global, a malloc result, a local. (The cartoon says: text < data < bss < heap < stack.)

c
/* lab1.c */
#include <stdio.h>
#include <stdlib.h>

int g_init = 42;      /* data */
int g_zero;           /* bss  */

int main(void) {
    int local = 7;
    int *heap = malloc(sizeof *heap);

    printf("text  (main)    %p\n", (void *)main);   /* fn-ptr-to-void* is a POSIX-ism; fine on Linux */
    printf("data  (g_init)  %p\n", (void *)&g_init);
    printf("bss   (g_zero)  %p\n", (void *)&g_zero);
    printf("heap  (malloc)  %p\n", (void *)heap);
    printf("stack (local)   %p\n", (void *)&local);

    puts("\n--- /proc/self/maps ---");
    FILE *f = fopen("/proc/self/maps", "r");
    for (int c; (c = fgetc(f)) != EOF; ) putchar(c);
    fclose(f);
    free(heap);
    return 0;
}
sh
gcc -O0 -g -o lab1 lab1.c && ./lab1

Expected: (numbers WILL differ because of ASLR, but the pattern holds)

text  (main)    0x5586a3f2a189
data  (g_init)  0x5586a3f2d010
bss   (g_zero)  0x5586a3f2d01c
heap  (malloc)  0x5586a4a012a0
stack (local)   0x7ffd8c1a4a54

--- /proc/self/maps ---
5586a3f2a000-5586a3f2b000 r-xp ... /work/lab1
5586a3f2d000-5586a3f2e000 rw-p ... /work/lab1
5586a4a01000-5586a4a22000 rw-p ... [heap]
7f0e3d600000-7f0e3d7c1000 r-xp ... /usr/lib/x86_64-linux-gnu/libc.so.6
7ffd8c185000-7ffd8c1a6000 rw-p ... [stack]

Compare: find each printed address inside a maps range. main falls in the r-xp mapping of your binary (text), the globals in its rw-p mapping, the malloc'd pointer inside [heap], the local inside [stack]. The cartoon's ordering holds, and at the same time you can see it is a cartoon: dozens of mappings, not five, and every run at different addresses. Run it twice and diff.

Lab 2: see padding and endianness in raw bytes§

Predict first: struct { char c; long l; } with c='A' (0x41) and l=0x12345678. Write out the 16 bytes you expect, remembering 7 padding bytes and little-endian order.

c
/* lab2.c */
#include <stdio.h>
#include <string.h>

struct demo { char c; long l; };

int main(void) {
    struct demo d;
    memset(&d, 0xAA, sizeof d);   /* paint everything, so padding is visible */
    d.c = 'A';
    d.l = 0x12345678;
    printf("sizeof = %zu\n", sizeof d);
    fwrite(&d, sizeof d, 1, stderr);   /* raw bytes on stderr */
    return 0;
}
sh
gcc -O0 -o lab2 lab2.c && ./lab2 2>&1 >/dev/null | hexdump -C

Expected:

00000000  41 aa aa aa aa aa aa aa  78 56 34 12 00 00 00 00  |A.......xV4.....|
00000010

Read it: 41 is 'A'; the seven aa bytes are the padding, visible only because we painted them; then 78 56 34 12 00 00 00 00 is the 8-byte long 0x12345678, little end first. The standard doesn't guarantee padding contents, so gcc/clang here just left our paint alone. One dump, both concepts. Now reorder the struct as { long l; char c; }, predict again, and re-run.

Lab 3: trigger the classic bugs and read the fault§

Predict first: what does a 1 MiB write into a 16-byte stack buffer do?

c
/* lab3.c */
#include <string.h>
int main(void) {
    char buf[16];
    memset(buf, 'A', 1 << 20);   /* 1 MiB into 16 bytes */
    return 0;
}
sh
gcc -O0 -o lab3 lab3.c && ./lab3; echo "exit: $?"

Expected:

Segmentation fault (core dumped)
exit: 139

That's the raw experience: the overflow marched toward higher addresses (memset fills upward) until it ran off the high end of the mapped [stack] region and the hardware faulted. Exit 139 = killed by signal 11 (SIGSEGV). Now get a real diagnosis by recompiling with AddressSanitizer:

sh
gcc -O0 -g -fsanitize=address -o lab3-asan lab3.c && ./lab3-asan

Expected: a report beginning like

==NN==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f... 
WRITE of size 1048576 at 0x7f... thread T0
    #0 ... in main /work/lab3.c:5

You get the exact bug class, address, size, file, and line. Then do the same for use-after-free: compile the 5-line UAF example from the chapter body ("Bug 2") without ASan and watch it exit cleanly, silently corrupting as it goes, then with ASan and get heap-use-after-free with both the free and allocation stacks. That contrast (silent success vs. instant diagnosis) is the most important thing this lab teaches.

Three labs in, you can take a live address and say which region owns it, predict a struct's size before the compiler tells you, and turn a bare Segmentation fault into a bug class with a file and a line. Chapter 11 builds page tables on top of the first of those.

Bridge notes§

If you've had compiler-design and microprocessor coursework: sections on the byte array, pointers, endianness, stack frames, and struct padding are known material, so skim the tables to confirm the x86_64 numbers and move on. Worth your 15 minutes anyway: Lab 1's /proc/self/maps correlation. Plenty of people who can draw the cartoon have never matched live pointers against real VMA lines, and the twice-run ASLR diff is a good habit. Likely genuinely new even for you: the glibc specifics (M_MMAP_THRESHOLD / M_TRIM_THRESHOLD at 128 KiB, free-list pointers stored inside freed user data, the mechanism that makes UAF exploitable rather than merely wrong); the standard's "pointer becomes indeterminate at lifetime end" wording (see WG14 N2443, "lifetime-end pointer zap," for how deep that rabbit hole goes); and the caveat that padding bytes need not survive member stores.

Sources§