📜 C and the Machine
The C language as it actually runs on hardware: the build pipeline, types, pointers, memory layout, the ABI, allocation, and undefined behavior.
1.1 Why C, and how to read this chapter
C exists because systems programmers needed a language that could describe real machines without being trapped in assembly for every new processor. Dennis Ritchie developed it at Bell Labs in the early 1970s, in the same environment that produced Unix, and that ancestry still matters. The language is small enough that an experienced programmer can hold much of it in their head, but it is also sharp enough to express the things operating systems and devices actually care about: bytes, addresses, object layout, calling conventions, volatile hardware state, and code that must run before a comfortable runtime exists.
That is why C remains the common language of kernels, drivers, embedded firmware, packet-processing libraries, and performance-sensitive networking code. The Linux kernel is written in C, using a particular GNU dialect, and its networking and driver interfaces are exposed through C structures, function pointers, macros, and memory-ordering rules. Kernel-bypass systems such as AMD Solarflare Onload still live in this world: even when the application calls a familiar sockets API, the performance story depends on user-space stacks, driver support, DMA buffers, cache behavior, and careful control over what code runs on the data path.
The useful mental model is not that C is portable assembly. It is a thin, standardized language over machine realities. The standard gives names to objects, types, expressions, storage duration, and undefined behavior; the implementation maps those rules onto instructions, registers, stack frames, alignment constraints, and memory accesses. Good C programmers move between those two views constantly. They ask both "what does the source say?" and "what must the compiler, CPU, operating system, and device be allowed to do?"
That distinction is where many interview-grade bugs hide. A uint32_t field may look like a register, but the compiler still has rules about ordinary objects. A pointer may print like an address, but it also carries type and provenance assumptions. A struct may look like a wire header, but padding, alignment, and byte order decide whether the bytes are the bytes you meant. A fast path may be only a dozen lines of C, yet its real cost can be a cache miss, a branch misprediction, a syscall boundary, or a misplaced load from a descriptor ring.
The rest of this chapter builds the vocabulary for reading such code without superstition. It starts at translation from source to executable form, then tightens the model around representation, addresses, process memory, aggregate layout, calls, allocation, optimizer assumptions, and the practical toolchain. The goal is not to memorize trivia about C. The goal is to look at low-level code and reason, precisely, about what state exists, who owns it, how it is represented, and what machine actions the program can legitimately cause.
Sources
1.2 The compilation pipeline
A C program does not become a program in one step. The command cc main.c -o main is a driver command: it coordinates several tools and passes inputs from one stage to the next. In the traditional Unix model those stages are preprocessing, compilation, assembly, and linking. With GCC-like toolchains, expect cpp for the preprocessor, cc1 for the C compiler proper, as for the assembler, and ld for the linker, although modern drivers may integrate some of these internally.
The input to the front of the pipeline is not merely the bytes in one .c file. C is compiled in translation units. A translation unit is what remains after a source file has had its preprocessing directives handled: included headers are expanded, macros are expanded, conditional compilation is applied, and the resulting token stream is presented to the compiler as one unit of analysis. This is why a header is not normally compiled by itself; its contents become part of each .c file that includes it.
After preprocessing, the compiler no longer sees a neat file with #include lines. It sees the declarations and macro expansions pulled in from headers, plus the remaining code from the source file. Stopping after this stage with -E produces preprocessed C text. That output is useful when macros or generated headers make the real compiler input hard to see.
The compiler proper takes the translation unit and performs syntactic and semantic analysis: it checks that the token stream is valid C, builds internal representations, reasons about declarations and uses, and emits target-specific assembly language. Stopping here with -S commonly produces a .s file. This assembly is still text, but it is no longer C. It names instructions, registers, labels, and assembler directives. The compiler has translated C into a lower-level representation, but many addresses are still unknown.
The assembler turns assembly text into an object file, usually a .o file. On Linux, the usual format is ELF, the Executable and Linkable Format. At this point the file is binary, but normally not executable. It is a relocatable object: it contains machine code and data in sections, plus metadata that lets another tool combine it with other objects. A source file that calls printf, or a function defined in another .c file, cannot know that function's final address while it is being assembled. The assembler emits placeholders and records what must be fixed later.
Two pieces of metadata make this work: symbols and relocations. A symbol is a named thing the toolchain may need across boundaries: a function, an object with storage, or a label-like location. An object file has a symbol table describing symbols it defines and symbols it references but does not define. A relocation entry says, in effect, "at this offset in this section, adjust the encoded value once the final address of this symbol is known." An "undefined reference" means the linker could not find a definition for a referenced symbol. A "multiple definition" means it found competing definitions where the rules require one.
The linker consumes relocatable object files and libraries. Its job is to collect needed pieces, resolve symbol references, apply relocations, lay out output sections, and produce a new ELF file. That output may be an executable, a shared object, or another relocatable object. This is the first stage that sees the set of objects and libraries being linked. Separate compilation works because each .c file can become a .o independently, and linking later stitches those partial results together.
Libraries are packaging mechanisms for object code. A static library, conventionally libname.a, is an archive of relocatable object files, historically made with ar. When the linker searches a static library, it pulls in archive members needed to satisfy currently unresolved symbols. The selected code becomes part of the linked output. That can simplify deployment in firmware, small utilities, and tightly controlled low-level components, but updating the library requires relinking the program.
A shared library, conventionally libname.so on Linux, is itself an ELF shared object. When you link against it, the executable records that it needs the shared object and includes dynamic-linking information for runtime resolution. At startup, the dynamic linker/loader, such as ld.so or ld-linux.so, finds the needed shared objects, maps them, performs required relocations, resolves dynamic symbols according to platform rules, and transfers control so the program can run. This matters in systems work: a packet stack, vendor library, or diagnostic tool may be deployed either statically or through compatible shared objects on the target system.
The pipeline is therefore a sequence of increasingly concrete artifacts. Preprocessing produces the source text the compiler will analyze. Compilation produces assembly. Assembly produces relocatable object files with symbols and relocation records. Linking produces the loadable program image or shared object, after resolving the names and address fixups that could not be settled earlier. Many build failures are simply failures at a particular boundary, caused by the information that stage does or does not yet have.
Sources
1.3 Types, integers, and representation
C gives you integer types, not a promise that every machine uses the same integer sizes. The standard fixes ordering constraints and minimum ranges: short and int are at least 16 bits, long is at least 32 bits, and long long is at least 64 bits. It does not say that int is always 32 bits, though that is common on server and NIC systems.
The exact choices form a platform's data model. On common Unix-like 64-bit targets, including typical x86-64 and AArch64 Linux systems, the model is LP64: long and pointers are 64 bits, while int remains 32 bits. Older or embedded 32-bit ABIs are often ILP32: int, long, and pointers are 32 bits. Windows 64-bit C uses LLP64: long long and pointers are 64 bits, but long stays 32 bits. That is why portable systems code does not use long to mean "machine word", "pointer-sized integer", or "protocol field". Those are three different ideas.
For low-level work, the distinction matters because hardware and wire formats have exact widths. A NIC register field documented as bits 31:16 is not an int; it is part of a 32-bit register image. An IPv4 address, a queue producer index, or a DMA descriptor word should usually be expressed with a type whose width says what the interface says, such as uint32_t, not with a type whose size follows the host ABI.
Every standard integer type has signed and unsigned forms, but they are not just "with negatives" and "without negatives". Unsigned arithmetic is arithmetic modulo one more than the largest representable value. For an N-bit unsigned type, results are reduced modulo 2^N. That makes masks, shifts, counters, and ring indices natural in unsigned types:
uint32_t next = (prod + 1u) & (ring_size - 1u);
uint32_t flags = word & 0x0000ff00u;
Signed integers represent negative and positive values. On current mainstream C implementations, signed integers use two's complement representation: an N-bit signed type has values from -2^(N-1) through 2^(N-1)-1, and the same bit pattern can be interpreted as either signed or unsigned depending on the type. Older C standards allowed other signed representations, but production systems code has overwhelmingly lived on two's complement machines for decades. The useful mental model is that the bits are real, but the type tells C which arithmetic rules apply to those bits.
The dangerous rule is overflow. Unsigned wrap is defined. Signed overflow is undefined behavior. If int x = INT_MAX; x + 1 is evaluated, the C abstract machine does not define a wrapped result. Optimizers use that rule when transforming comparisons and loops. If you want modulo arithmetic, use an unsigned type of the intended width. If you want checked signed arithmetic, test before the operation or use checked-arithmetic helpers where available.
C also silently changes integer types before many operations. Integer promotion happens first for small integer types such as char, signed char, unsigned char, short, unsigned short, and narrow enum types. If int can represent all values of the original type, the value is promoted to int; otherwise it is promoted to unsigned int. This is why arithmetic on uint8_t usually happens as int, not as uint8_t:
uint8_t a = 250, b = 10;
int sum = a + b; /* commonly 260, after promotion */
uint8_t low = a + b; /* converted back, commonly 4 */
After promotion, many binary operators apply the usual arithmetic conversions to choose a common type. If both operands have the same type, nothing more happens. If both are signed or both are unsigned, the lower-rank operand converts to the higher-rank type. Mixed signed and unsigned arithmetic is where interview bugs and production bugs both appear. If the signed type can represent every value of the unsigned type, the unsigned operand converts to signed; otherwise both convert to an unsigned type. A common consequence is that len - header_len may become a huge unsigned value if len is unsigned and the subtraction underflows.
The <stdint.h> header gives you a vocabulary for saying what you mean. The exact-width types int8_t, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t, and uint64_t exist only when the implementation provides matching types with exactly that width and no padding bits; on ordinary modern targets they exist. The least types, such as uint_least32_t, promise at least a width. The fast types, such as uint_fast32_t, let the implementation choose an efficient type with the minimum width. intptr_t and uintptr_t are optional integer types capable of holding converted pointer values; use them only when you truly need the pointer-as-integer operation discussed in the next section.
The practical rule is simple: use plain int for small local counts where the range is naturally modest and signed arithmetic is desired; use size_t for object sizes and allocation extents; use fixed-width unsigned types for registers, descriptors, masks, and protocol fields; and be deliberate at signed/unsigned boundaries. Most integer bugs in systems C are not caused by not knowing the CPU. They are caused by asking C to infer a width and signedness at exactly the point where the hardware interface required precision.
Sources
1.4 Pointers and memory addresses
A pointer is a typed value that designates an object, a function, or a special null value. In ordinary systems language we often say "a pointer is an address", and that is a useful first approximation: int *p can hold the address of an int, and *p accesses the int there. But the C type matters. It tells the compiler what kind of object may be accessed, how large one element is for arithmetic, and what result type a dereference has.
int x = 42;
int *p = &x;
*p = 43;
&x forms a pointer to x. *p is the object reached through p. The variable p itself is also an object with storage; it contains a pointer value. Changing p changes where it points; changing *p changes the object it points at.
Addresses live in an address space. On real machines they are represented by bits in registers and memory, but C does not define pointers as plain integers. Low-level code may receive addresses from hardware descriptions, memory maps, or DMA setup paths, but turning such values into C pointers is implementation- and platform-specific.
Pointer arithmetic is scaled by the pointed-to type. If p has type T *, then p + n means "the element n positions after p", not "add n bytes". The byte displacement is conceptually n * sizeof *p.
uint32_t words[4];
uint32_t *p = words;
p + 1; /* next uint32_t, not next byte */
The standard model for this arithmetic is an array. If p points at element i, p + 1 points at element i + 1, and p - q gives the distance in elements when both pointers refer into the same array. A pointer one past the last element may be formed and compared, but not dereferenced.
for (uint32_t *it = words, *end = words + 4; it != end; ++it)
consume(*it);
This is why bounds are not decoration in C. A packet buffer pointer and a length belong together. Advancing a header pointer through a received frame is meaningful only while the result still denotes positions in the same buffer.
Arrays and pointers are closely related but not identical. An array object contains its elements; a pointer object contains a pointer value. In most expressions, an array expression is converted to a pointer to its first element. That is why a[i] and *(a + i) name the same element, and why a function parameter written as uint8_t buf[] is adjusted to uint8_t *buf.
uint8_t a[64];
uint8_t *p = a; /* a converts to &a[0] */
sizeof a; /* size of the whole array */
sizeof p; /* size of the pointer object */
The exceptions matter. The array does not decay as the operand of sizeof, unary &, or string-literal initialization of a character array. Confusing an array with a pointer causes wrong sizes, especially in helper functions that receive a buffer parameter and then attempt sizeof buf.
void * is C's generic object pointer. Any object pointer can be converted to void * and back, preserving the value. This is how callback interfaces carry an untyped context pointer:
void poll_rx(void *arg)
{
struct rx_queue *q = arg;
service_queue(q);
}
A void * does not point to an object of known size or representation, so standard C does not allow dereferencing it or doing arithmetic on it. Convert it first. GNU C extends the language by allowing arithmetic on void * as if the element size were 1; portable C should use unsigned char * or uint8_t * for byte-wise movement.
Function pointers are pointers too, but they are not object pointers. A function pointer designates callable code and is called through the pointed-to function type:
int (*cmp)(const void *, const void *);
int r = cmp(a, b);
This syntax reads inside out: cmp is a pointer to a function taking two const void * arguments and returning int. Driver code uses this pattern for operation tables, callbacks, and completion handlers. Standard C keeps function pointers and object pointers distinct; do not assume that a function pointer can be stored in void * portably.
Const correctness describes what may be changed through a name. const int *p means p points at an int that must not be modified through p; the pointer may still be advanced or reassigned. int * const p means the pointer object p itself is const after initialization, but the int it points at may be modified. const int * const p makes both promises.
const int *a; /* pointer to const int */
int * const b = &x;/* const pointer to int */
For interfaces, put const where it communicates ownership and mutation. A parser that only inspects packet bytes should accept const uint8_t *buf; a routine that fills a transmit descriptor cannot. This lets the compiler reject accidental writes through read-only views and makes callback signatures honest.
Sources
1.5 The memory layout of a process
When you run a C program, the executable is not copied into one flat array of RAM. The kernel creates a process with a virtual address space: a private map from program addresses to physical pages managed by the kernel and MMU. For now, treat every pointer value you print as a virtual address inside this process.
On a typical x86-64 Linux process, low to high addresses look roughly like this:
- executable mappings:
.text,.rodata,.data,.bss - the heap, growing upward as the program break moves with
brkorsbrk mmapmappings, used for shared libraries, file mappings, anonymous mappings, thread stacks, and often large allocations- the initial thread stack, placed high and growing downward
Exact addresses vary because of ASLR, PIE, kernel policy, resource limits, and architecture. The useful invariant is what kind of object may live in each region and what protection the kernel applies.
The .text region holds machine instructions. It is normally readable and executable, not writable. The .rodata region holds read-only data: string literals, jump tables, const objects whose addresses are needed, and constants selected by the compiler and linker.
The .data region holds objects with static storage duration that have explicit nonzero initial values:
int link_speed_mbps = 25000;
static unsigned rx_budget = 64;
Those initial bytes must exist in the file image, because the loader needs their starting values. The .bss region holds static-storage objects that are uninitialized or initialized to zero:
static unsigned dropped_packets;
char dma_scratch[4096];
C says these objects start as zero. ELF represents this efficiently: the loadable segment has a memory size larger than its file size, and the extra bytes are defined as zero. So a 1 MiB zeroed static buffer contributes to the process image without storing 1 MiB of zero bytes in the executable file.
Above the static image is the heap. Historically, the C allocator could extend it by moving the program break upward; Linux exposes this through brk and sbrk. Modern allocators also use mmap, especially for large allocations. Allocator internals are later; here the point is that dynamically allocated objects are not in .data or .bss. They live in memory obtained at run time.
The mmap area is a collection of mappings: shared objects such as libc.so, file-backed mappings, anonymous memory, guard pages, and sometimes allocator-managed regions. mmap creates a virtual mapping and can map files or devices, which is why the same mechanism matters when low-level code exposes device memory to a process.
Near the top is the stack for the initial thread. On x86-64 Linux it grows downward: more stack use consumes lower addresses. Stack frames, call boundaries, register save rules, and the System V AMD64 ABI come next.
The executable file is organized for two audiences. Linkers work mostly with sections such as .text, .rodata, .data, and .bss. The loader works mostly with segments, described by ELF program headers. A PT_LOAD segment says: map this range of file bytes at this virtual address with these permissions, and if the in-memory size exceeds the file-backed size, zero the rest. Several sections can end up inside one loadable segment. The process runs the virtual mappings constructed from the ELF program headers, plus the interpreter, shared libraries, stack, heap, and other mappings.
Alignment is another constraint on bytes. A scalar object usually has a natural alignment: an int wants an address divisible by its alignment, a pointer wants pointer alignment, and wider vector or atomic objects may demand more. Hardware can make misaligned loads slower, split them across cache lines, or reject them. The compiler therefore places objects at suitable addresses and inserts padding inside structs when needed. This is why sizeof(struct x) can exceed the sum of its fields. The detailed rules belong in the structs section; the principle is simple: layout includes invisible bytes inserted to keep later fields aligned.
Dynamic allocation preserves the same rule. Standard malloc returns storage suitably aligned for any standard object type with fundamental alignment. DMA rings, cache-line ownership protocols, SIMD buffers, and device descriptors often need stronger alignment, so driver and packet-processing code uses posix_memalign, page allocation, huge pages, or kernel DMA allocation interfaces.
Endianness is the order of bytes within a multi-byte integer. In little-endian order, the least significant byte is stored at the lowest address. In big-endian order, the most significant byte is stored first. x86-64 is little-endian, and common ARM/AArch64 systems are normally run little-endian, although ARM supports endian choices. Internet protocols define network byte order as most-significant byte first, so code uses htons, htonl, ntohs, and ntohl at protocol boundaries.
This matters anywhere bytes cross a boundary: packet headers from the wire, memory-mapped device registers, descriptor rings read by a NIC, and DMA buffers shared with hardware. C lets you name memory, but it does not make layout, alignment, permissions, or byte order disappear. Low-level correctness begins with knowing which bytes are where and how the other side will interpret them.
Sources
1.6 Structs, unions, and bitfields
A struct is C's way to describe a record: several objects laid out as one larger object. The standard guarantee is deliberately simple. Non-bit-field members have addresses that increase in declaration order, and a pointer to the structure, suitably converted, points at its initial member. The compiler may insert unnamed padding between members so that each member begins at a suitably aligned address. It may also insert trailing padding after the last member, because an array of struct T must make every a[i] correctly aligned.
That last rule is why sizeof is not just the sum of member sizes. On ordinary ABIs, a structure's alignment is at least the strictest alignment of any member, so char, uint64_t, char usually wastes space compared with uint64_t, char, char. Reordering fields by decreasing alignment often shrinks descriptor, flow-table, or packet-metadata structures, but field order can also encode an ABI, a hardware format, cache locality, or readability.
struct rx_meta {
uint64_t dma_addr;
uint32_t len;
uint16_t flags;
uint8_t queue;
};
Use the language tools to state and check layout, not comments. sizeof x includes trailing padding. offsetof(struct rx_meta, flags), from <stddef.h>, gives the byte offset of a named non-bit-field member. C11 added _Alignas and _Alignof; many codebases use <stdalign.h>'s alignas and alignof names, and C23 makes those spellings keywords. These matter when a DMA-visible object, cache-line-separated counter, or MMIO shadow must start at a particular boundary. If layout is a contract, assert it:
_Static_assert(sizeof(struct rx_meta) == 16, "rx_meta ABI changed");
_Static_assert(offsetof(struct rx_meta, flags) == 12, "bad flags offset");
A union is different: its members overlap. All members begin at the same address; the union is large enough for its largest member and aligned strictly enough for any member. In C, if you store through one union member and read through another, the bytes are reinterpreted as the read member's type; the resulting value may be implementation-defined, unspecified, or even a trap representation. That idiom is not the same as freely casting arbitrary pointers. Reading object bytes through unsigned char * is special and valid for inspection, but casting a packet buffer to struct iphdr * and dereferencing it may violate alignment, effective-type, or aliasing assumptions. Section 9 returns to the optimizer side of that bargain.
The common initial sequence rule is precise. If a union contains several structures whose first members have compatible types, and matching bit-fields have the same widths, C permits inspecting that shared initial part through any of those structure members while the completed union type is visible. This lets tagged alternatives put the tag at offset zero without duplicating storage.
union event {
struct { uint16_t type; uint16_t len; } h;
struct { uint16_t type; uint16_t len; uint32_t rxq; } rx;
struct { uint16_t type; uint16_t len; uint32_t txq; } tx;
};
Bit-fields let a structure or union member occupy a named number of bits, normally inside an allocation unit of some integer type:
struct desc_flags {
unsigned own : 1;
unsigned csum_ok : 1;
unsigned vlan_present : 1;
};
They are convenient for local flags, but dangerous as binary formats. C leaves major details implementation-defined: whether a plain int bit-field is signed, which bit is allocated first within a unit, whether a field may straddle an allocation-unit boundary, and what nonstandard base types are allowed. You also cannot apply & to a bit-field, and offsetof is not valid for one. The source does not portably say which wire bit became vlan_present.
For hardware registers and protocol headers, prefer explicit masks on fixed-width integers unless the target ABI and compiler are part of the contract:
#define DESC_OWN (1u << 0)
#define DESC_CSUM_OK (1u << 1)
uint32_t flags = load_desc_flags();
if (flags & DESC_OWN)
process_desc();
This is the same discipline that keeps packet code honest. A C struct describes a host object layout; an Ethernet, IPv4, TCP, or device descriptor format describes bytes at specified offsets, often with specified byte order. Those are not automatically the same thing. A struct overlay can be reasonable for an MMIO register block, but the pointer should normally be to volatile-qualified objects because each register access is an observable hardware interaction. Even then, volatile does not fix ordering, atomicity, or cache coherency; it only constrains ordinary compiler elision and merging.
The GNU and Clang extension __attribute__((packed)) tells the compiler to reduce or remove padding. It is useful when matching a documented external format, but it also reduces member alignment, so accessing a naturally aligned member may compile to slower byte operations or fault on targets that reject unaligned loads. Combining packed with explicit alignment is common in driver code, but treat it as an ABI declaration, not a performance trick.
The robust pattern is deliberate: define constants for offsets and bit masks; memcpy bytes into fixed-width temporaries; convert byte order; check sizes and offsets with _Static_assert; and reserve struct overlays for places where the ABI and hardware manual are under control. NIC descriptors, MMIO register blocks, and packet headers reward that discipline because one padding byte can turn correct-looking C into a wrong bus transaction.
Sources
1.7 The stack, calls, and the ABI
The stack is the process region used to make function calls nest. Treat it as a LIFO of activation records, or stack frames. If main calls poll_loop, which calls rx_batch, which calls parse_ipv4, each live call needs somewhere to keep the state it will resume with when its callee returns. Recursion is the same problem with many instances of the same function alive at once. On x86-64, pushing moves the stack pointer toward lower addresses; popping moves it upward.
A stack frame is not a C object. It is ABI-shaped memory plus registers. At minimum, an ordinary x86-64 call pushes the return address, the address of the next instruction in the caller. A later ret pops that address and transfers control back there. Around it the compiler may place saved registers, locals that did not stay in registers, spill slots, alignment padding, and outgoing arguments that do not fit in argument registers.
The current top of stack is named by rsp. Many functions adjust rsp in their prologue to reserve frame space and restore it in their epilogue. rbp can be a stable base, or frame, pointer: save old rbp, copy rsp into rbp, then address locals and arguments at fixed offsets. But rbp is optional. Optimizing compilers may omit it, for example with -fomit-frame-pointer, and use unwind metadata instead.
The hardware gives you call, ret, rsp, and registers. The ABI tells separately compiled code how to use them together. On Linux, BSD, macOS, and other Unix-like x86-64 systems, C code follows the System V AMD64 ABI. For integer and pointer arguments, the first six are passed in rdi, rsi, rdx, rcx, r8, and r9. Further integer or pointer arguments are passed on the stack. The first eight floating-point or SSE-class arguments use xmm0 through xmm7.
Return values have matching conventions. Ordinary integer and pointer returns come back in rax; a 128-bit integer return uses rdx:rax. Floating-point returns use xmm0. Large aggregates can involve hidden pointers, but the simple register cases are the ones you constantly see in hot-code disassembly.
long add3(long a, long b, long c)
{
return a + b + c;
}
For a System V AMD64 call to add3, a arrives in rdi, b in rsi, and c in rdx; the long result leaves in rax.
The same contract divides registers into caller-saved and callee-saved sets. A caller-saved register is scratch across a call: if the caller needs its old value later, it must spill it before the call and reload it after. A callee-saved register must be preserved by the called function: if the callee uses it, the callee saves the incoming value and restores it before returning.
Under System V AMD64, the callee-saved registers are exactly rbx, rbp, rsp, and r12 through r15. The rest, including rax, rcx, rdx, rsi, rdi, and r8 through r11, are caller-saved scratch. This split is why a compiler can put short-lived temporaries in rax or r10 across straight-line code, but must be more careful when emitting a call. It is also why hand-written assembly cannot clobber rbx and return: the caller is entitled to find its old rbx still there.
Two stack details matter constantly in low-level work. First, System V AMD64 requires 16 bytes stack alignment at a call boundary: just before the call, the caller arranges alignment so that on entry to the callee, after call has pushed the 8-byte return address, rsp + 8 is a multiple of 16. This lets callees use aligned stack slots for vector spills and keeps compiler-generated code interoperable.
Second, System V AMD64 defines a 128-byte red zone below rsp. A leaf function, one that makes no calls, may use that space without moving rsp. This is a user-space optimization, not a hardware fact. Do not rely on the red zone in interrupt or signal entry code, kernel code, or contexts built with -mno-red-zone; Windows x64 has no red zone. Windows x64 is a different ABI: its first integer or pointer arguments use rcx, rdx, r8, and r9, and the caller reserves 32 bytes of shadow space. Same ISA, different platform contract.
That is why the ABI matters. It is the reason a C object file from one compiler can call a library built by another, why Rust, Go, Python extensions, and debuggers can agree on an extern "C" boundary, and why syscall wrappers have precise register shuffling rules. It is also why assembly fast paths, interrupt and exception entry stubs, context switches, and setjmp/longjmp must know exactly which state is live and who owns saving it. In a NIC driver or packet loop, these rules explain why the seventh argument appears at a stack offset, why a leaf checksum helper might not subtract from rsp, why a crash backtrace can unwind one function but not another, and why clobbering one spare-looking register can corrupt a caller many frames away.
Sources
1.8 Dynamic memory and allocators
The stack gives storage with lexical lifetime; malloc gives storage whose lifetime is controlled explicitly. The C interface is small: malloc(size) reserves at least size bytes and leaves them uninitialized, calloc(n, size) reserves n * size bytes and zero-initializes them, realloc(p, size) changes an allocation and may move it, aligned_alloc(alignment, size) asks for specified alignment, and free(p) releases a block.
The contract is narrow. A successful allocation returns a pointer suitably aligned for any object type with a fundamental alignment requirement. That is why malloc(sizeof *p) can be assigned to a struct foo * without an alignment shim. If malloc(0) is called, C permits either a null pointer or a non-null pointer that must not be dereferenced but can be passed to free. free(NULL) is a no-op. Any other pointer passed to free must be a live pointer returned by malloc, calloc, realloc, or aligned_alloc; freeing a stack address, an interior pointer, a pointer already freed, or a pointer from a different allocation API is undefined behavior.
realloc often exposes sloppy ownership thinking. If it succeeds, the old pointer is no longer the handle unless the returned value compares equal; if it fails, the original allocation is still live, so assign through a temporary.
Real allocators manage larger regions obtained from the operating system and carve them into chunks. A chunk normally has allocator metadata near the user-visible payload, including size and state information. Boundary-tag allocators store enough size information to find neighboring chunks, so adjacent free chunks can be coalesced. That is why a write just past the end of your buffer is so poisonous: it may not fault immediately, but it can scribble on the next chunk's header and make a later free or malloc operate on corrupted metadata.
glibc's allocator is in the ptmalloc family, derived from Doug Lea's dlmalloc ideas and adapted for multithreaded programs. Freed chunks are organized by size and recent history: fast bins for very small recently freed chunks, small bins, large bins, and an unsorted bin used as a staging area before chunks are sorted. The allocator also keeps a top chunk, the wilderness at the end of an arena that can be split or extended. In threaded programs, glibc uses multiple arenas so not every allocation contends on one global lock; by default the arena limit is related to the number of online CPU cores, with a 64-bit default cap of eight times that count unless the tunable is set.
Small and medium allocations usually come from arena-managed heap memory, which can grow through the program break interface. Large allocations may bypass the normal heap and be served directly with mmap, so the allocator can return the mapping to the kernel on free. In glibc the glibc.malloc.mmap_threshold tunable defaults to 131072 bytes, 128 KiB, when unset, and glibc may adjust that threshold dynamically.
Alignment matters beyond correctness. On GNU systems, malloc and realloc return blocks aligned to 8 bytes on 32-bit systems and 16 bytes on 64-bit systems; on x86-64 that is 2 * sizeof(void *). That is enough for ordinary fundamental-alignment C objects. It is not a promise of cache-line, hugepage, or DMA-friendly alignment. For stronger alignment, use posix_memalign(&p, alignment, size) on POSIX systems or C11 aligned_alloc(alignment, size), remembering that C11 requires the size argument to be a multiple of the alignment. The resulting pointer, when allocation succeeds, is still released with free.
Fragmentation is the tax paid for variable lifetimes and sizes. External fragmentation means enough free memory exists in total, but it is split into pieces too small or inconveniently placed for the next request. Coalescing and binning are allocator countermeasures. Internal fragmentation means the allocator gave you a larger chunk than requested because it rounded up for metadata, alignment, or a size class; the slack is inside your allocation and cannot satisfy another request.
The classic heap bugs are ownership bugs made concrete. A memory leak happens when the program loses the last usable pointer to a live allocation. A double-free releases the same allocation twice; in glibc-style allocators that can corrupt free-list metadata and has historically been an exploitation primitive, but the C-level fact is simpler: it is undefined behavior. A use-after-free dereferences a pointer after the allocation's lifetime ended; it may appear to work until the chunk is reused. A heap buffer overflow writes outside the allocated extent and may corrupt another object or the allocator's headers.
Setting a pointer to NULL immediately after free(p) is useful local discipline because a repeated free(p) then becomes free(NULL). It is not a full solution: aliases still exist. In low-level networking code, the stronger discipline is structural: clear ownership rules, fixed-size object pools, freelists, or slab-style caches for descriptors and packet buffers. Hot paths avoid steady-state malloc/free because allocator locks, arena selection, page acquisition, and fragmentation add latency variance. When buffers need cache-line or page alignment, or must meet DMA mapping constraints, allocate them explicitly with the right alignment and lifetime.
AddressSanitizer and Valgrind's Memcheck catch these bugs. The toolchain chapter returns to them; the rule here is simpler: the allocator is part of your program's correctness boundary.
Sources
1.9 Undefined behavior and the optimizer
Dynamic allocation gave us a way to ask the runtime for storage. Undefined behavior is the other side of C’s bargain with the machine: direct, cheap access to storage and arithmetic, but only inside a contract. When the contract is broken, the standard does not say “do whatever this CPU happens to do.” It says there are no requirements on the implementation for that execution.
This differs from two nearby ideas. Implementation-defined behavior is chosen by the implementation and documented. Unspecified behavior is a choice among valid possibilities where the implementation need not document which one it picked each time; function-call argument evaluation order is the usual example. Undefined behavior is not a documented choice and not a portable machine primitive. It is outside the program the compiler must preserve.
The optimizer matters because C is specified through an abstract machine and the “as-if” rule: the compiler may emit any code whose observable behavior matches the abstract execution. Observable behavior includes volatile accesses and I/O, not every load or store you imagined. If a path would execute undefined behavior, the compiler may reason that a correct C program never reaches it. From there it can delete checks, hoist loads, or fold branches.
int grow(int x)
{
if (x + 1 < x)
return -1;
return x + 1;
}
On ordinary hardware, adding one to the maximum signed int may wrap, trap, or set flags. In C, signed integer overflow is undefined. The optimizer may assume x + 1 does not overflow in any defined execution, making x + 1 < x false. The check can disappear. Unsigned arithmetic is defined modulo 2^N, so wraparound on uint32_t is part of the contract. Flags such as -fwrapv ask a particular compiler for signed wrapping semantics, but that is a toolchain choice, not the default C rule.
Strict aliasing is another contract that surprises low-level programmers because it conflicts with the “all memory is bytes” mental model. C lets you inspect object representation through character types: a char *, signed char *, or unsigned char * may access the bytes of any object. But accessing an object’s stored value through an incompatible non-character lvalue is generally undefined. The standard describes this with the object’s effective type. Compilers use it for type-based alias analysis: if an int * and a float * point into the same apparent address, optimized code may assume they do not name the same int object.
That matters in packet and driver code. A received Ethernet frame is naturally a byte buffer, but casting an arbitrary uint8_t * to a protocol struct * and reading fields may combine alignment, effective-type, padding, and endian bugs. The robust idiom is to copy bytes into a typed object with memcpy, then decode fields explicitly. Optimizers understand small fixed-size memcpy, so this is usually correct and fast. Unions are sometimes used for type punning; C has allowances when access is through the union type, and GCC documents cases, but pointer casts that bypass the union object often fall back into undefined behavior.
Sequencing rules are the contract for side effects inside expressions. Older C texts speak of sequence points; C11 reformulated this as evaluations being sequenced before, unsequenced, or indeterminately sequenced. The engineering rule is simple: do not both modify a scalar object and read or modify it again in an unsequenced way.
i = i++; /* undefined */
a[i] = i++; /* undefined */
f(i++, i); /* argument order unspecified */
In the first two cases, the side effect of i++ and another use or modification of i are unsequenced relative to each other. There is no “compiler chose left-to-right” answer. In the function call, the arguments are evaluated before the call begins, but their relative order is unspecified; if the expressions do not create an unsequenced conflict, the program is valid, just not pinned to one order.
Finally, volatile is narrower and more useful than folklore says. An access through a volatile-qualified lvalue is an observable side effect. The compiler must issue the access according to the abstract machine; it may not elide it, merge repeated volatile register reads into one, or reorder volatile accesses relative to other visible side effects in ways the abstract execution would not allow. That is why volatile belongs on memory-mapped device registers and on the limited volatile sig_atomic_t style of communication with a signal handler.
It is not a concurrency primitive. volatile does not make a read-modify-write atomic. It does not make a shared ring index safe between producer and consumer cores. It does not flush store buffers, invalidate cache lines, or create the acquire/release ordering a NIC driver needs around descriptors. For inter-thread and inter-core communication, use atomics, locks, barriers, and device accessors. volatile tells the compiler that this access is externally observable; it does not tell the CPU or memory system how to order the rest of the world.
Sources
1.10 The toolchain in practice
The compiler is not a black box you invoke at the end of the Makefile. For systems work, it is part of the machine you are programming. You need to know which contract you are asking it to honor: easy debugging, fast code, small code, stronger diagnostics, or runtime checking.
gcc and clang are largely command-line compatible for ordinary C builds, because clang implements a GCC-compatible driver interface. Optimization decisions and diagnostics differ, but the working vocabulary is shared: -O2 -g -Wall -Wextra, -S, -fsanitize=address, and so on.
The optimization level is the first big switch:
-O0is the default when no-Ooption is given. It keeps compilation fast and makes source-level debugging behave most like the text you wrote.-O1enables a conservative set of optimizations intended to reduce code size and execution time without spending much compile time.-O2enables nearly all supported optimizations that do not usually make a space-speed tradeoff. This is a common release baseline for C systems code.-O3adds more aggressive transformations such as extra loop optimizations and inlining. It can be faster, but can also make code larger or stress instruction cache locality.-Osoptimizes for size, enabling most-O2optimizations except ones that often increase code size.-Ogis meant for the edit-compile-debug cycle: some optimization is enabled, but passes that badly damage the debugging experience are avoided.
-g asks the compiler to emit debugging information, usually DWARF on modern Unix-like systems. It does not turn optimization off. You can build with -O2 -g, and often should for a release-only bug, but the debugger may show optimized reality: variables may be unavailable, statements may appear reordered, and some source expressions may not exist as separate machine operations. For first-pass debugging, -Og -g is usually saner than -O0 -g; for performance inspection, use the level you intend to ship.
Warnings are cheap static scrutiny. -Wall does not mean “all warnings”; it means a broad set the compiler authors consider generally useful and avoidable. -Wextra adds another layer. -Werror turns warnings into build failures, which is excellent for controlled codebases, but awkward with third-party code or changing compiler versions.
When performance or hardware-visible behavior matters, read the generated assembly. The fastest path is:
cc -O2 -g -S -fverbose-asm rx.c
objdump -d rx.o
objdump -drwC -Mintel rx.o
-S stops after producing assembly. -fverbose-asm asks GCC to add helpful comments. objdump -d disassembles object code or executables, showing what the assembler actually encoded. On x86, GNU tools default to AT&T syntax: source before destination, %rax, $1, and suffixes such as movq. Intel syntax reverses operands and omits much punctuation; objdump -M intel -d requests it. The syntax matters less than the discipline: confirm whether a bounds check disappeared, whether a load stayed inside a loop, or whether an MMIO access compiled into the width and ordering you intended.
gdb is the microscope for a running or crashed user-space process. Start with gdb ./prog, set a breakpoint with break main or break file.c:42, then run. Use next to step over calls, step to enter them, continue to resume, and finish to run until the current function returns. When stopped, bt gives a backtrace; frame N selects a caller; print expr evaluates a C expression in the current frame. For raw state, x/16gx ptr examines memory as sixteen giant words in hexadecimal, info registers prints registers, and disassemble shows instructions around a function or address.
Core dumps are the postmortem version. If the OS writes a core file, gdb ./prog core lets you inspect the crashed state: stack, registers, memory mappings, and local variables if debug info survived.
Sanitizers are compiler-inserted runtime checks. They change the binary, cost CPU and memory, and are not production hardening. Their value is that they catch failures at the point of corruption, before the bad pointer or data race becomes a meaningless crash ten thousand packets later.
- AddressSanitizer, enabled with
-fsanitize=address, detects out-of-bounds accesses to heap, stack, and globals, use-after-free, and related addressability errors. Leak detection is provided through LeakSanitizer and is enabled by default with ASan on Linux. - UndefinedBehaviorSanitizer, enabled with
-fsanitize=undefined, instruments selected undefined behaviors such as signed integer overflow, invalid shifts, null misuse, and misaligned accesses. Its overhead is usually much smaller than ASan or TSan. - ThreadSanitizer, enabled with
-fsanitize=thread, detects data races. It is powerful but expensive: Clang documents typical slowdown around5xto15xand memory overhead around5xto10x.
A useful development build is often -Og -g -Wall -Wextra -fsanitize=address,undefined; run -fsanitize=thread separately because race detection has different requirements and overhead. For packet handling, descriptor rings, and buffer ownership, this pays quickly. A one-byte overrun in a receive buffer or a stale pointer after recycling a packet object can pass unit tests until the data path is hot enough to corrupt something unrelated.
The habit is simple: compile with intent, read what the compiler produced, debug the state the machine actually has, and run instrumented builds before trusting uninstrumented speed.
Sources