⚡ High-Performance Networking
Where the standard stack runs out: why it's slow, interrupt mitigation and polling, zero-copy and batching, kernel bypass (DPDK, Onload/ef_vi), XDP, RDMA, and tail latency.
9.1 When the kernel stack isn't enough
The default kernel network stack is the right starting point for most programs. It gives you routing, congestion control, firewalling, namespaces, observability, fair sharing, and a stable sockets API. The moment you call recv() or send(), though, you have chosen a general-purpose path: packets are represented as kernel objects, moved through queues, classified, accounted, copied or mapped, woken through scheduling machinery, and finally delivered to an application thread. That machinery is valuable, but it is not free.
At low packet rates, the cost is hidden by useful work. At high packet rates, the unit of pain becomes the packet, not the byte. A 100 Gb/s link carrying 1500-byte frames is demanding; the same link carrying minimum-size Ethernet frames is brutal. With 64-byte frames, the wire rate is about 148.8 Mpps, which leaves only a few CPU cycles per packet on a single core before line rate is mathematically impossible. Every cache miss, branch, allocation, interrupt, lock, copy, and kernel/userspace transition becomes visible.
The first ceiling is latency. A packet may spend more time waiting behind software work than crossing the network. Interrupt delivery, softirq scheduling, socket-buffer allocation, protocol processing, wakeups, and application scheduling add variance as well as mean delay. For trading, storage, RPC fanout, telemetry ingestion, and distributed training control paths, the 99.9% tail can matter more than the median: one unlucky queueing delay can stall an otherwise fast system.
The second ceiling is throughput. A NIC can DMA descriptors and packet buffers at enormous rates, but the host must keep its receive rings drained and transmit rings filled. If the driver cannot replenish buffers fast enough, or the CPU cannot retire per-packet work fast enough, drops appear even though the optical link is fine. This is where NIC and driver details stop being implementation trivia: queue count, RSS hashing, interrupt moderation, DMA mapping, descriptor layout, NUMA placement, cache locality, checksum offload, TSO/GSO, and receive-side batching decide whether the machine behaves like a network endpoint or a packet furnace.
Leaving the default path is therefore not an aesthetic choice. It is a trade: keep the kernel stack when its services are worth the overhead; move work earlier, batch it harder, copy less, poll instead of interrupting, or bypass the kernel when the overhead dominates the useful computation. The rest of this chapter follows those pressure points: why the conventional path costs what it costs, how Linux mitigates that cost, and why systems such as DPDK, Onload, ef_vi, XDP, eBPF, and RDMA exist.
Sources
9.2 Why the kernel stack is slow
The kernel network stack is not slow because it is careless. It is slow, for high packet-rate work, because it is general-purpose. It must protect processes, multiplex sockets over devices, implement TCP/IP semantics, enforce routing and firewall policy, collect statistics, support tracing, and survive hostile input. Each requirement is reasonable. The problem is that packets are small, so fixed costs can dominate useful payload work.
Start at the application boundary. A normal socket program receives data with recvmsg, recv, or read, and transmits with sendmsg, send, or write. Each call crosses from user mode into kernel mode. That is a privilege transition: the CPU enters a controlled kernel path, validates arguments, touches kernel socket state, and returns to user mode. It is not necessarily a process context switch, because the same thread may enter and leave the kernel without another process running. But it is still a repeated boundary crossing.
The per-call cost matters because packets are numerous. On small messages, one syscall per packet can cost more than the application's own parsing. High-performance interfaces therefore amortize boundary crossings with batching: sendmmsg, recvmmsg, io_uring, DPDK bursts, and NIC receive rings all attack the same fixed-cost problem.
Copies are the next obvious cost. On receive, the NIC DMAs packet bytes into memory owned by the kernel or driver. The kernel builds metadata around the packet, usually a struct sk_buff, runs protocol processing, and copies payload into the user buffer. On transmit, the application gives the kernel a user buffer; the kernel commonly copies or references that data in socket buffers before the driver arranges DMA to the NIC. Copies burn memory bandwidth, consume cache capacity, and add latency proportional to payload size. Linux has mechanisms such as MSG_ZEROCOPY and sendfile to reduce copying in some cases, but the ordinary socket interface is built around a protection boundary between user memory and kernel memory.
Interrupts are another fixed cost. In the simplest model, every received packet causes the NIC to interrupt a CPU. The CPU stops what it was doing, enters interrupt handling, acknowledges the device, and schedules network processing. That model collapses at high rates: the machine spends its time being interrupted rather than draining the receive queue. Linux NAPI changes the shape of the cost. The device interrupt schedules a NAPI poll instance; the driver then processes packets in a polling loop, commonly with receive interrupts suppressed for that queue until the work is drained or the budget is exhausted. Many interrupts become fewer interrupts plus batches of packet work.
NAPI does not make packet processing disappear. It moves much of it into software interrupt context, commonly called softirq processing. Softirqs defer expensive work out of the hard interrupt handler while still running promptly. But they create scheduling pressure. If softirq work is heavy, it can run before returning to user mode or be pushed into ksoftirqd, a kernel thread that competes with application threads. A busy receive path can steal CPU time from the process that is trying to consume the data. This is one reason tail latency can degrade before average throughput looks bad.
Blocking and wakeups add another layer. If an application calls recv on an empty socket, it may sleep. When a packet later arrives, the kernel enqueues data, marks the task runnable, and the scheduler eventually runs it. That is a real context switch: registers, stack, scheduler state, and often cache working set change from one execution stream to another. Even nonblocking designs can trigger context switches indirectly when worker threads or ksoftirqd take CPU time.
The less visible cost is cache and locking behavior. A packet is represented by NIC descriptors, driver state, an sk_buff, protocol headers, route and neighbor state, socket queues, memory accounting, filter state, and application buffers. Each step touches memory. If the NIC interrupt is handled on one CPU, TCP processing runs on another, and the application thread runs on a third, cache lines migrate between cores. Coherent multicore systems must transfer ownership of modified cache lines, and a contended line can become a throughput limiter.
Locks and atomic operations compound the problem. Socket receive queues, qdisc state, reference counts, backlog queues, allocator metadata, and statistics all require coordination. Modern Linux networking works hard to shard this state: multiqueue NICs, Receive Side Scaling, Receive Packet Steering, per-CPU counters, and careful queue ownership all exist because shared state is expensive. But a general stack cannot assume one queue, one flow, one core, and one application. A driver or kernel-bypass dataplane often can.
This is the interview-level model: the kernel path pays for generality one packet at a time. Syscalls cross protection domains. Copies cross memory domains. Interrupts and softirqs move work into asynchronous kernel contexts. Scheduler decisions move execution between threads. Cache misses, cache-line bouncing, atomics, and locks turn a short code path into a shared-system problem. NIC and driver engineers care because many wins come from changing where those costs occur: coalescing interrupts, choosing queue-to-CPU affinity, batching descriptors, preserving cache locality, avoiding unnecessary sk_buff work, and deciding when the kernel stack is the wrong dataplane abstraction.
Sources
9.3 Interrupt mitigation and polling
The simplest receive path is also the easiest one to overload: every arriving packet makes the NIC raise an interrupt, the CPU stops what it was doing, enters the interrupt handler, acknowledges the device, and starts moving packets from the receive ring into the network stack. That model is responsive at low rates. At high rates it becomes self-defeating. Even after realistic framing, one interrupt per packet is not a scheduling mechanism; it is a denial-of-service attack against the CPU.
Interrupts are expensive because they disrupt locality and control flow. The core takes an asynchronous trap, saves state, runs privileged code, touches device or MSI-X state, and often wakes softirq processing. The packet still has to be DMA-synchronized, described by metadata, classified, and delivered upward. If the interrupt rate grows with the packet rate, the machine spends more time reacting to work than doing it.
Interrupt coalescing attacks the first part of that problem in hardware. Instead of interrupting immediately for each receive completion, the NIC waits until a threshold is reached: perhaps N packets have arrived, or a timer of M microseconds has expired after the first packet in a batch. Linux exposes many of these knobs through ethtool -c and ethtool -C, with names such as rx-usecs, rx-frames, tx-usecs, and tx-frames; the exact set is driver and NIC dependent.
The tradeoff is fundamental. Coalescing improves throughput and CPU efficiency because one interrupt can announce many completions. It also adds waiting time. If rx-usecs is set to 50, the first packet in a burst may sit in the NIC ring for up to roughly that timer interval before the CPU is told about it. That is fine for bulk TCP or storage traffic, but bad for workloads where extra microseconds become visible in tail latency.
Linux NAPI is the software half of the same idea, but it is more subtle than "turn interrupts off and poll forever." NAPI uses a hybrid interrupt-then-poll model. When a receive interrupt fires, the driver's interrupt handler does a small amount of work: it acknowledges or masks the relevant interrupt source and schedules a struct napi_struct with napi_schedule() or a related helper. The real packet draining happens later in the driver's NAPI poll method.
The poll method is given a budget. It should process up to that many receive packets from the NIC's DMA ring, replenish receive buffers, and usually also reap transmit completions. If it consumes the whole budget and still has work, it returns the budget, and the kernel will poll it again without another hardware interrupt. If it drains the available work, it calls napi_complete_done() and the driver unmasks receive interrupts. Under load, the device stops interrupting for every packet; the CPU keeps pulling batches until pressure drops.
This is why NAPI is central to NIC driver work. The driver must get interrupt masking right, or it will either lose wakeups or livelock. It must respect the receive budget, or one busy queue can starve other queues and softirq work. It must order DMA descriptor reads, buffer recycling, and device doorbells correctly, or polling will race the NIC. Modern multiqueue NICs replicate this machinery per queue, often with one MSI-X vector, NAPI instance, ring, and CPU affinity per queue.
NAPI also changes the way drops happen. When the host cannot keep up, packets can accumulate in the NIC receive ring and be dropped by hardware before the kernel spends cycles allocating and freeing per-packet objects. That is not "good" in an application sense, but it is better than receive livelock, where the CPU burns all its time taking interrupts for packets it cannot process.
At the opposite end from interrupt coalescing is busy polling. A thread that cares about minimum latency can spin, repeatedly checking for receive work instead of sleeping until an interrupt or scheduler wakeup arrives. In Linux sockets this appears through SO_BUSY_POLL and the net.core.busy_read and net.core.busy_poll sysctls, when the kernel is built with the needed support. The socket layer can poll the associated NAPI context for a bounded time while a blocking receive or poll()/select() waits for data.
Busy polling buys latency by spending CPU. It avoids interrupt delivery latency, scheduler latency, and cross-CPU wakeup effects, but the core is burning cycles while there may be no packet to receive. That can be right for a dedicated low-latency service pinned to an isolated CPU, especially when traffic is frequent. It is usually wrong where power and fair CPU sharing matter.
These techniques form a continuum. Per-packet interrupts minimize idle latency but collapse under high packet rates. Interrupt coalescing and NAPI batch work to protect throughput and stability. Busy polling spends a core to remove wakeup delay. Interview questions hide in concrete details: when does the driver mask the IRQ, what does budget mean, and why can adaptive coalescing improve throughput while hurting p99 latency? The answer is always the same shape: packets arrive in hardware rings, and every transition from device to CPU has a cost.
Sources
9.4 Zero-copy and batching
The fastest byte is the one the CPU does not move. A normal read followed by write looks innocent in C, but the machine work is substantial: storage or the NIC DMA engine places data in kernel memory, the kernel copies it into a user buffer, then a send path copies it back into kernel-owned socket buffers before the NIC can DMA it out. Each copy consumes memory bandwidth and pollutes caches. At 100 Gbit/s, payload traffic is roughly 12.5 GB/s; one extra full copy is already a serious part of the memory bandwidth budget.
Zero-copy means arranging ownership and addressability so that data can move between devices and the network stack without redundant CPU copies. It does not mean “no data movement”: the NIC still performs DMA, cache coherency still costs, and metadata is still created. The goal is to avoid copying unchanged payload bytes.
sendfile is the classic file-server example. Instead of read(fd, buf, n) then send(sock, buf, n), the application asks the kernel to transfer bytes from an input file descriptor to an output file descriptor. On Linux, the common use is file-to-socket. The kernel can keep data in the page cache and attach those pages to socket buffers, avoiding the user-space bounce buffer.
Mapped buffers apply the same idea from the other direction. With mmap, a file or device-backed region is mapped into the process address space, so the application can touch memory also meaningful to the kernel or device subsystem. Packet sockets have PACKET_RX_RING and PACKET_TX_RING, where application and kernel share a memory-mapped ring. Modern zero-copy socket transmit, such as Linux MSG_ZEROCOPY, pins user pages and lets the network stack build transmit descriptors that refer to those pages. The hard part is lifetime: the application must not overwrite a buffer until completion says it is free.
That lifetime rule is where low-level driver thinking enters. A NIC transmit ring does not contain packets; it contains descriptors: “DMA this address, this length, with these flags.” If the address points at pinned application pages or page-cache pages, transmit saves memory bandwidth, but the driver, IOMMU, DMA mapping layer, and completion path must preserve correctness. Receive is harder because the destination buffer must exist before the packet arrives.
Copy avoidance often exposes the next fixed cost: doing one operation at a time. A packet send involves a system call boundary, descriptor allocation, protocol accounting, queue selection, possible DMA mapping, and a doorbell write to the NIC. Some costs are nearly independent of packet size. Paying them once for 64 bytes is much worse than paying them once for a group of packets or a larger segment.
Batching amortizes those fixed costs. sendmmsg and recvmmsg let an application transmit or receive multiple datagrams with one system call. writev and sendmsg gather multiple buffers into one logical send without first concatenating them. TCP segmentation offload and generic segmentation offload let upper layers work with a larger logical packet while hardware or lower software later cuts it into MTU-sized frames. On receive, GRO coalesces related packets so the upper stack sees fewer, larger units. In kernel-bypass APIs, the same pattern is explicit: receive a burst, process a burst, enqueue a burst.
The arithmetic is simple. If a fixed operation costs C cycles and per-packet work costs P cycles, one-at-a-time processing costs C + P per packet. A batch of N packets costs roughly C / N + P per packet, until cache pressure, queueing, or lock contention changes the curve. This is why packet code cares about ring sizes, refill thresholds, and doorbell batching. A driver that rings the NIC doorbell after every descriptor burns PCIe transactions; one that rings once for many descriptors usually gets higher throughput.
The trade-off is latency, especially tail latency. A batch has to form. If packets are already arriving at high rate, it fills naturally and the added delay can be tiny. If traffic is sparse, waiting for N packets may add avoidable microseconds or milliseconds. Even without an explicit timer, batching can hide in queues: a socket send buffer, an interrupt-moderation interval, or an event loop that drains many completions before returning to one latency-sensitive request.
Good systems therefore batch by budget: “up to N packets,” “up to X bytes,” or “until a short time budget expires.” Bulk replication, telemetry export, and packet forwarding can tolerate larger batches; RPCs, market-data responses, and control-plane messages often prefer smaller batches or explicit flush points. The useful instinct is to ask what fixed cost is being amortized and what queueing delay is introduced in exchange.
Zero-copy and batching are not magic switches. For small messages, pinning pages and processing completions can cost more than a copy. For large or sustained flows, copies waste memory bandwidth and batching raises packets per second per core. NIC, driver, and kernel work lives in that boundary: descriptor formats, buffer ownership rules, completion semantics, and thresholds that keep the fast path full without hiding latency-sensitive work behind bulk traffic.
Sources
9.5 Kernel bypass: DPDK
DPDK moves the packet hot path out of the kernel and into a user-space process. The application is no longer asking the kernel to receive a packet, allocate an sk_buff, classify it through the socket layer, and wake a thread. Instead, a DPDK process owns NIC queues through a Poll Mode Driver, or PMD, and repeatedly asks the device: are there completed receive descriptors yet? Are there transmit descriptors I can reclaim? This is still ordinary DMA hardware. The difference is where the rings are managed.
A PMD is a user-space driver for a class of NICs. During initialization, DPDK discovers devices, maps PCI resources, configures ports and queues, and arranges memory that the NIC can DMA to and from. At run time the fast path is small: receive a burst from an RX queue with rte_eth_rx_burst(), inspect or modify packets, then enqueue a burst to a TX queue with rte_eth_tx_burst(). The driver polls descriptor rings directly rather than relying on per-packet interrupts. Link-status interrupts may still exist, but packet I/O is designed around polling.
That design spends CPU to buy determinism. A DPDK forwarding core may look busy even when traffic is light, because it is spinning rather than sleeping. Under load, however, it avoids interrupt entry and exit, scheduler latency, wakeups, socket locks, and generic kernel bookkeeping. DPDK does not make PCIe DMA, cache misses, or descriptor write-backs disappear; it makes them visible to the application and removes unrelated layers from the critical path.
The packet object is usually an rte_mbuf. It carries metadata such as data length, packet length, port, offload flags, RSS hash results, VLAN information, and chained-buffer pointers. The bytes live in buffers allocated from an rte_mempool: a pool of fixed-size objects, normally created once at startup and reused for the life of the data plane. Dynamic allocation on the packet path adds locks, cache disruption, unpredictable latency, and failure modes that appear only under pressure.
Hugepages are the other half of the memory story. Linux normally uses 4 KiB pages. A high-rate packet application can touch a large buffer area; with small pages, the CPU burns cycles on TLB misses, and the IOMMU has more translations to manage. DPDK therefore uses hugepage-backed memory, commonly 2 MiB pages and sometimes 1 GiB pages, for packet buffers and shared data structures. On NUMA machines, buffers for a NIC attached to one socket should usually come from memory local to that socket, and the polling core should usually run there too.
Mempools also have per-lcore caches. If every packet allocation and free updated one shared global pool, the application would replace kernel contention with user-space contention. A core-local cache lets an lcore allocate and free mbufs in batches, touching shared pool state less often. This is why mempool size, cache size, RX descriptor count, and burst size matter: they determine shared-state traffic, burst absorption, and whether the NIC starves for receive buffers.
The simplest DPDK program is a run-to-completion loop. One lcore owns one RX queue and usually one TX queue. It polls, processes, and transmits before returning to the top of the loop:
for (;;) {
struct rte_mbuf *pkts[32];
uint16_t n = rte_eth_rx_burst(port, queue, pkts, 32);
for (uint16_t i = 0; i < n; i++)
handle_packet(pkts[i]);
uint16_t sent = rte_eth_tx_burst(port, queue, pkts, n);
for (uint16_t i = sent; i < n; i++)
rte_pktmbuf_free(pkts[i]);
}
Real code must handle drops, output selection, checksums, and offload flags. The loop works in bursts because descriptor work, function-call overhead, prefetching, and PCIe-visible updates amortize better over several packets.
Run-to-completion also explains why DPDK deployments isolate cores. A polling lcore is meant to own its queue, mempool cache, cache working set, and ideally its CPU time. If the Linux scheduler moves unrelated work onto that core, or migrates the DPDK thread, locality collapses. Production systems commonly combine DPDK lcore masks, CPU affinity, IRQ affinity, NUMA-aware memory allocation, and sometimes kernel boot parameters that keep housekeeping work away from data-plane cores. The goal is a narrower latency distribution, not just high average throughput.
The programming model is therefore closer to writing a small packet-processing operating system than writing a socket application. The Environment Abstraction Layer initializes CPU, memory, PCI, timers, logs, and process support. The Ethernet device API configures ports, RX queues, TX queues, RSS, offloads, MTU, and promiscuous mode. The application owns packet lifetime: allocate from a mempool, hand buffers to RX, receive filled mbufs, decide their fate, transmit them or free them. A leak drains the pool; an early free can corrupt data still visible to the NIC or another core.
For a low-level NIC interview, the important point is that DPDK exposes the hardware-shaped problem. You have descriptor rings with producer and consumer indexes, DMA buffers whose physical or IOVA addresses must be meaningful to the device, cache lines bouncing between cores, and device writes arriving over PCIe. DPDK is not magic acceleration; it is a disciplined way to remove general-purpose kernel costs so the remaining bottlenecks are the real data-plane bottlenecks.
Sources
9.6 Kernel bypass: Onload and ef_vi
DPDK asks the application to become a packet-processing system: own cores, queues, buffers, and usually a custom networking model. Solarflare Onload, now AMD Onload through the Solarflare/Xilinx lineage, takes a different route. It is a user-level TCP/UDP stack that accelerates the normal BSD sockets API. A dynamically linked program can often be run under onload without source changes, because Onload intercepts libc socket calls with LD_PRELOAD and routes eligible sockets through a user-space stack with direct, protected access to the NIC datapath. The application still calls socket, send, recv, poll, and epoll; the fast path is what changes.
That transparency is the engineering tradeoff. Onload keeps TCP/IP interoperability and a familiar sockets contract, so it fits existing trading gateways, collectors, caches, and RPC services more easily than a raw-packet framework. But it is not "the NIC implements TCP." The protocol stack runs in the process. The win comes from avoiding kernel crossings, socket locks, scheduler interactions, and shared kernel queues on the hot path. The kernel still handles setup, protection, routing, fallback, and reset recovery.
Under Onload sits ef_vi, short for EtherFabric virtual interface. Where Onload gives you sockets, ef_vi gives you a layer-2 API: raw Ethernet frames in registered user buffers, plus receive and transmit rings. An ef_vi program must build or parse Ethernet, IP, UDP, or other headers itself. This is natural for UDP market-data receive, timestamping, capture, or a custom feed handler; it is much less attractive for general TCP unless you are prepared to implement a real transport.
The central object is a VI, a virtual interface. A VI is not a Linux netdev; it is a per-client NIC datapath endpoint. Allocating one consumes hardware resources such as receive and transmit descriptor rings, an event queue, timers, interrupt state, and doorbell mappings. Receive filters steer selected traffic to that VI by MAC, VLAN, protocol, IP address, or port depending on adapter and firmware support. This is where NIC design meets isolation: the adapter must DMA only to memory in the protection domain, deliver matching traffic, and let many stacks share a port.
The receive path is deliberately close to a driver receive path. The application allocates packet buffers, registers memory so the NIC may DMA to it, posts receive descriptors, and rings a doorbell. When a matching frame arrives, the NIC writes packet bytes into one buffer and appends a receive event. The application polls the event queue, decodes the event, processes the packet, and reposts the buffer. The kernel has moved out of the steady-state data path, but ownership, queue depth, memory ordering, and buffer lifetime still have to be correct.
Transmit is the mirror image. The application builds a complete Ethernet frame, excluding the FCS, in memory the NIC can read. It posts a transmit descriptor and notifies the adapter. Later, a transmit completion event says the buffer can be reused. Completions are not optional; without them, a high-rate program eventually reuses memory still owned by hardware or loses track of queue capacity.
The event queue is the unifying completion mechanism. A VI exposes typed events: receive complete, transmit complete, transmit error, timer, and timestamp variants. In a low-latency design, a core may spin in a tight poll loop over this queue, trading CPU for predictable wakeup. In a less aggressive design, interrupts or blocking waits may be used. Either way, the event queue must be sized consistently with the descriptor rings; if completions cannot be recorded quickly enough, the datapath can stall.
Solarflare adapters also expose transmit latency shortcuts. With normal DMA transmit, the host writes a descriptor, rings a doorbell, and the NIC later fetches packet bytes from host memory. TX_PUSH optimizes the empty-queue case by combining notification with descriptor data so the adapter can start sooner. It reduces command-path overhead, but detecting and exploiting the empty-queue case can add polling or ordering cost.
CTPIO, cut-through programmed I/O, goes further. Instead of waiting for the NIC to DMA the frame from host memory, the CPU streams the frame to the adapter over PCIe using programmed I/O, and the adapter can begin sending with very little buffering. In ef_vi, CTPIO is requested when allocating the VI, then sends use calls such as ef_vi_transmit_ctpio. Correct code still posts a fallback DMA descriptor with a registered-memory copy, because CTPIO can fail or be unsuitable for a particular send. The completion event is returned whether the frame used CTPIO or fallback DMA, preserving the same ownership discipline.
For NIC and driver work, Onload and ef_vi are a compact case study. The API may be sockets or raw frames, but the hardware questions remain: How are queues allocated? How is memory registered and protected? Which cache lines does the hot path touch? What does the doorbell write order guarantee? How are filters programmed? What happens on reset? Kernel bypass removes the kernel from the steady-state packet path; it does not remove the driver/hardware contract.
Sources
9.7 XDP and eBPF fast paths
XDP sits at an interesting point in the packet path: it is still Linux, still using the kernel's driver model, but it lets a small eBPF program run at the earliest practical receive hook. In native mode, the NIC driver calls the XDP program from its receive path, typically while draining an RX ring under NAPI and before allocating an sk_buff. That placement matters. An sk_buff carries the Linux networking contract: protocol state, routing, netfilter, socket delivery, and many corner cases. It is expensive if the answer is simply "drop it", "forward it", or "send it to user space."
An XDP program receives an xdp_md context. From that context it gets packet bounds such as data and data_end; the verifier requires every packet access to be proven within those bounds. Before reading an Ethernet, IPv4, or TCP header, the program checks that it fits. The verifier, JIT compiler, and helper-call model make it acceptable to load this logic dynamically into the kernel without treating it like an arbitrary kernel module.
The program returns a small action code:
XDP_PASSmeans continue into the ordinary network stack. The driver will build or arrange ansk_buff, and the packet proceeds toward GRO, routing, netfilter, and sockets.XDP_DROPmeans discard the packet immediately. This is the classic high-rate filtering case: bad traffic can be rejected before socket lookup, conntrack, or allocation pressure.XDP_REDIRECTmeans send the frame somewhere else through a BPF map-backed target, such as another netdev, CPU map, devmap, or AF_XDP socket.
There are also XDP_TX, which transmits the received frame back out the same interface, and XDP_ABORTED, an exceptional failure path. But pass, drop, and redirect capture the main design: decide the packet's fate while it is still a raw frame in a driver-owned buffer.
A minimal filter has the shape below. The discipline matters more than the policy: parse only after bounds checks, then return an action.
SEC("xdp")
int drop_non_ipv4(struct xdp_md *ctx)
{
void *data = (void *)(long)ctx->data;
void *end = (void *)(long)ctx->data_end;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > end)
return XDP_DROP;
if (eth->h_proto != bpf_htons(ETH_P_IP))
return XDP_DROP;
return XDP_PASS;
}
For NIC and driver work, the subtle part is where this code runs relative to hardware rings. The NIC has DMA-written packet bytes into receive buffers described by RX descriptors. The driver's poll routine sees completed descriptors, constructs an XDP buffer view, and invokes the BPF program before committing to the normal stack. If the result is XDP_DROP, the driver can recycle the buffer. If it is XDP_PASS, the driver pays the stack cost. If it is XDP_REDIRECT, the driver must cooperate with redirect machinery and flush the redirect batch. Capabilities such as redirect, ndo_xdp_xmit, AF_XDP zero-copy, hardware offload, or non-linear XDP buffers are advertised per netdev.
AF_XDP extends this model into user space without giving the application complete ownership of the NIC in the DPDK sense. An AF_XDP socket, often called an XSK, is bound to a netdev and queue. User space creates a UMEM region divided into frames and shares rings with the kernel: a fill ring, an RX ring, a TX ring, and a completion ring. An XDP program redirects selected frames into the socket through an XSKMAP, commonly using bpf_redirect_map().
In copy mode, the kernel copies packet data between driver buffers and the AF_XDP UMEM. In zero-copy mode, when the driver and NIC support it, the NIC can DMA directly into UMEM-backed frames, and ownership moves by ring descriptors rather than payload copies. That sounds close to full bypass, but the boundary is different. The kernel still owns the netdev, verifies the XDP program, manages the socket, arbitrates queue binding, and handles traffic passed to the ordinary stack. AF_XDP is a selective fast lane attached to Linux, not a replacement operating environment for the NIC.
This explains where XDP fits against full kernel bypass. DPDK or ef_vi is appropriate when an application wants direct ownership of queues, polling policy, memory pools, and most of the data-plane contract. XDP is appropriate when the system still benefits from Linux networking but needs early decisions: DDoS filtering, load balancing, telemetry sampling, container networking, L2/L3 forwarding, or steering hot flows into user space. It can drop before allocation, redirect before socket lookup, and preserve the kernel stack for ordinary traffic.
The price is that XDP programs are constrained. They run in a hot driver path, so they cannot block, allocate arbitrary memory, or perform unbounded work. Shared state normally lives in BPF maps, with the usual care about per-CPU data, atomics, cache locality, and update rates. For an interview, connect the abstraction back to the receive ring: XDP is fast because it makes a small, verified decision while the packet is still close to the NIC descriptor that delivered it. It is not fast because eBPF is magic; it is fast because it avoids general-purpose work for packets that do not need it.
Sources
9.8 RDMA and the verbs model
RDMA starts from a simple observation: if two machines already know which buffers are involved, the data path does not have to look like read(), kernel protocol work, socket-buffer copies, an interrupt, a wakeup, and then write() into the final destination. Remote Direct Memory Access lets a NIC move bytes directly between application memory on two hosts. The CPU still sets up the operation and handles completion, but the steady-state transfer is performed by the adapter.
The verbs model is the common programming abstraction for this. A "verb" is an operation submitted to an RDMA-capable device: create a queue, register memory, post a send, post an RDMA write, poll for completion. In Linux, libibverbs is the userspace API layered over kernel RDMA support, and ib_uverbs exposes controlled direct access to the hardware. This is kernel bypass in a precise sense: the application can ring hardware doorbells and poll completion queues without a syscall per packet. The kernel still creates objects, enforces access control, pins memory, maps device queues, and tears resources down.
The central object is the queue pair, or QP. A QP contains a send queue and a receive queue. The application posts work requests; the NIC consumes them asynchronously. A two-sided message uses both ends explicitly: the receiver posts buffers, the sender posts a send work request, and the hardware matches the message to a posted receive. If the receiver has not posted enough receives, the protocol can stall or fail. That receive-credit discipline is why RDMA applications pre-post many receives and replenish them from the completion path.
Completions are reported through completion queues, or CQs. A CQ is a producer-consumer ring written by the NIC and read by software. A work completion says that a posted operation has finished, usually carrying a status, byte count, identifier, and sometimes immediate data. Submission and completion are decoupled: a thread can post many work requests, ring one doorbell, and later poll a CQ for a batch of completions. That shape should look familiar to a NIC or driver engineer: descriptor rings, indexes, DMA ownership, memory ordering, and cacheline placement all matter.
RDMA becomes more distinctive with one-sided operations. With RDMA_WRITE, the initiator tells its NIC to write into a registered memory region on the remote host. With RDMA_READ, it fetches from remote registered memory into local registered memory. The remote CPU need not run code when data moves, and the remote application does not post a receive for each one-sided operation. The remote side first communicates the target virtual address, length, and remote key through some control path. Once the initiator has those capabilities, the NIC can perform the transfer subject to the region's permissions.
Memory registration is therefore not incidental setup; it is the mechanism that makes zero-copy RDMA safe enough to exist. When a process registers a buffer, the kernel pins the relevant pages so they cannot be paged out or relocated while the device may DMA to or from them. The RDMA subsystem programs translation and protection state into the adapter or IOMMU path, and returns keys such as lkey for local access and rkey for remote access. Work requests use address, length, and local key; remote one-sided requests also carry the remote address and remote key.
This explains the performance and cost model. Once memory is registered, payload bytes are not copied into kernel socket buffers. The NIC can DMA directly from source to destination. The application can submit work from userspace and poll completions without entering the kernel for each message. But registration itself is expensive: it pins pages, touches page tables, consumes device translation-cache resources, and creates lifetime constraints. Fast RDMA systems reuse registered buffer pools instead of registering per operation.
The verbs transport determines what the hardware promises. Reliable connected queue pairs give ordered reliable delivery and support RDMA read, write, send, and often atomics. Unreliable datagram modes look more like message datagrams. RoCE carries InfiniBand transport semantics over Ethernet, while iWARP maps RDMA over TCP-compatible transports. The programming model remains recognizable, but congestion control and loss behavior differ. At driver level, verbs objects correspond to hardware context: packet sequencing state, retry counters, protection domains, DMA mappings, and completion moderation behavior.
The hard part is often not issuing an RDMA_WRITE; it is building the protocol around it. One-sided writes do not by themselves tell the remote CPU that useful data arrived unless the design uses immediate data, a separate send, a polled flag, or a higher-level completion scheme. Registered memory must not be freed while the NIC can still access it. Remote keys must be treated as capabilities, because leaking an rkey plus address gives another endpoint permission to touch that memory region.
RDMA matters because it exposes the NIC as a programmable transport engine rather than a packet pump hidden behind sockets. The reward is very low CPU overhead and latency for storage, databases, MPI, and AI/HPC fabrics. The price is that application and driver code must manage resources the kernel TCP stack normally hides: buffer lifetime, flow control, registration pressure, queue depth, protection, and when hardware work is really complete.
Sources
9.9 Latency, throughput, and tail latency
Latency is time spent waiting for one unit of work. Throughput is work completed per unit time. They interact, but they are not substitutes. A link can carry 100 Gbit/s and still deliver a packet too late for a trading decision, storage completion, RPC fanout, or control-loop deadline. Conversely, a path can have excellent unloaded latency and poor throughput because it cannot keep enough work in flight.
Average latency lies by hiding shape. Suppose 999 packets complete in 5 us and one packet completes in 5 ms. The average is about 10 us, which sounds excellent, but one packet was delayed by three orders of magnitude. If that packet held a lock, carried a market data update, or released a storage queue entry, the system experienced the 5 ms event, not the average. Mean latency answers "how much total waiting was accumulated?" It does not answer "how bad can one important operation become?"
Percentiles preserve more of the distribution. The median, or p50, is the value at which half the observations were no worse. p99 means one observation in 100 was worse; p99.9 means one in 1000; p99.99 means one in 10000. In a service handling a million operations per second, a p99.99 event happens roughly 100 times per second. For a NIC receive path, those events may be packets arriving during an interrupt-disabled region, RX-ring refill, cache-line transfer, or firmware interrupt moderation interval.
Tail latency matters more when operations compose. If a request fans out to 100 back-end operations and must wait for all of them, the user sees the slowest one. A rare DMA mapping stall, a refill path that sometimes allocates memory, or a receive queue sharing cache lines with another core can become visible at the application boundary.
Jitter is variation in latency. In packet networks, the IETF defines packet delay variation using differences between one-way delays. In host networking work, people use jitter more broadly: any run-to-run variation in when work is observed or completed. Some jitter is inherent: frame serialization, switch pipelines, congestion, PCIe contention, DRAM state, cache state, TLB state, and branch predictor state. Much of it is host policy: interrupt moderation, NAPI budgets, scheduler migration, preemption, timer ticks, page faults, CPU frequency changes, lock contention, IOMMU misses, NUMA misplacement, ring overflow, and descriptor starvation.
Throughput can hide these effects. Batching is the classic example. Receiving 32 packets per poll amortizes MMIO, cache misses, and system calls, so packets per second improve. But the first packet waited while the rest arrived or while the poll loop reached it. Interrupt coalescing does the same at the hardware boundary. This is a valid trade; the mistake is reporting only throughput and mean latency, then being surprised when a deadline workload fails.
Honest measurement starts by defining the event. Is the start time when the application calls send(), when the packet reaches the driver, when the descriptor is posted to the NIC, when the first bit hits the wire, or when the peer application reads the data? Each boundary answers a different question. Driver work often needs software timestamps around the API, datapath timestamps around queues, and hardware timestamps near the MAC or PHY. Linux SO_TIMESTAMPING exists because software and hardware timestamps do not mean the same thing.
Use a monotonic clock for local intervals. Pin threads, warm caches, control CPU frequency, isolate test cores, bind IRQs and queues intentionally, and record enough samples. A p99.99 estimate from 10000 samples has only one sample beyond it; it is not stable. Histograms are usually better than averages because they preserve the shape.
Avoid coordinated omission. If a load generator sends the next request only after the previous response returns, then a 10 ms stall also prevents many would-have-arrived requests from being measured. The histogram shows one slow request instead of a period during which every scheduled request would have waited. For latency under load, drive requests according to the intended arrival process and record scheduled start time as well as actual start time. Closed-loop tests are useful, but they measure a closed-loop system, not independent arrivals.
Finally, measure at the load you care about and near saturation. Queueing delay is nonlinear: as utilization approaches capacity, small bursts and service-time variations produce large waits. A NIC path that looks perfect at 10% load may develop a tail at 80% because RX rings, cache, PCIe, and CPU budget are closer to exhaustion. Benchmark results must therefore include packet size, offered load, queue count, interrupt settings, CPU placement, NUMA placement, timestamp point, and loss policy. Without those details, a latency number is mostly a story about the harness.
Sources
9.10 AI and HPC fabrics
Distributed AI training and large HPC jobs turn the network into part of the computation. A GPU does not merely send a request and wait for a reply. During training, each worker computes gradients, then all workers must combine them before the next step. The common primitive is all-reduce: for k ranks, each rank contributes an array and every rank receives the reduced result. If one participant is late, the whole synchronized step is late.
The simplest mental model is a reduction followed by a broadcast, but real systems choose algorithms to fit message size, topology, and link speed. A tree all-reduce reduces up a tree and broadcasts back down. It has O(log k) steps, so it is attractive for small messages where per-step latency dominates. Its weakness is that upper-tree links and parent nodes carry more traffic than leaves. A ring all-reduce splits the buffer into chunks and moves them around a logical ring, usually as reduce-scatter followed by all-gather. Each rank sends to one neighbor and receives from one neighbor in each step. The ring takes O(k) steps, but for large buffers it can keep every link busy with balanced, pipelined transfers. This is why collective libraries care about topology discovery, GPU affinity, and rank placement.
These traffic patterns create congestion that looks different from ordinary client-server traffic. Incast occurs when many senders converge on one receiver or one switch output at nearly the same time. In AI and HPC fabrics it appears inside collective phases, parameter exchange, checkpointing, filesystem reads, and control fan-in. The hot output buffer is finite. If packets arrive faster than the egress link drains them, the queue grows; if it overflows, packets are dropped; if it is merely deep, latency and jitter rise. For synchronized jobs, that delay becomes a global barrier cost.
RDMA makes this more delicate. With reliable connected RDMA, the NIC implements much of the transport: queue pairs, packet sequence numbers, completion generation, and direct placement into registered memory. The CPU is not in the fast path, which is the point, but conventional TCP-style recovery is not what the application is relying on. A single loss can trigger NIC-level recovery, retransmission, timeout, or queue pair error behavior that is catastrophic compared with a few microseconds of normal fabric latency. The result is a strong preference for fabrics that avoid drops.
InfiniBand was designed as a lossless switched fabric. It uses link-level credit based flow control: a transmitter sends only when the receiver has advertised buffer space. That does not eliminate congestion, but it prevents normal data loss from buffer exhaustion and gives the fabric mechanisms for virtual lanes, service levels, congestion control, and adaptive routing. RoCE brings the InfiniBand transport model to Ethernet. RoCEv1 is Layer 2 Ethernet; RoCEv2 carries the InfiniBand transport over UDP/IP, commonly using UDP destination port 4791, so it can be routed as IP traffic while still exposing RDMA semantics.
Ethernet is normally lossy, so RoCE deployments often build a lossless Ethernet class for RDMA traffic. Priority-based Flow Control, standardized as IEEE 802.1Qbb, can pause one priority on a link instead of pausing all Ethernet traffic. RDMA can be protected while best-effort traffic continues. But PFC is a blunt local mechanism: one pause can propagate upstream and spread congestion. Misconfigured PFC can produce head-of-line blocking, unfairness, or deadlock.
ECN is the complementary signal. Instead of waiting until a queue overflows, a switch can mark congestion in the IP header using the ECN bits defined by RFC 3168. RoCEv2 congestion control can turn those marks into congestion notification packets, causing NICs to reduce injection rate before drops occur. PFC tries to prevent loss at the last moment; ECN tries to keep the queue from reaching that point. A well-tuned AI fabric depends on correct QoS classification and thresholds: too late and packets drop or pause storms form; too early and links sit idle.
GPUs stress the network because their local compute and memory systems are fast and bursty. A training step may have many GPUs finish a layer at almost the same time and immediately inject large gradient chunks. The data often moves GPU memory to NIC through PCIe, NVLink, or GPUDirect RDMA paths with little CPU pacing. That is ideal for throughput, but it removes smoothing that a slower software stack might have provided. The NIC sees large DMA reads and writes, deep work queues, and a requirement to sustain line rate while producing completions, honoring memory registration rules, applying congestion control, and preserving required ordering.
For low-level NIC and driver work, AI networking is not just faster packet I/O. The device is part of a distributed synchronization machine. Descriptor ring sizing, interrupt moderation, completion coalescing, PCIe behavior, receive buffers, PFC watchdogs, ECN handling, queue mapping, and telemetry counters all affect whether a cluster trains efficiently or stalls at the tail. At thousands of GPUs, a bad threshold becomes idle accelerators and unstable job time.
Sources