One byte's trip to the wire

1. The whole trip, in one view

One byte has to cross a dozen worlds to get from your program to the wire. Step the token through the map, then we'll zoom into every hop — instruction execution, address translation, caches, the doorbell, PCIe, the IOMMU, and the NIC's PHY.

The send path is a chain of ownership transfers. Software starts with a byte in a socket buffer, mbuf, or driver packet buffer, and ends with encoded symbols leaving a SerDes lane as voltage or light. In order, the byte crosses CPU instruction execution -> registers / store buffer -> caches and coherence -> DRAM / DMA buffers -> TX descriptor + MMIO doorbell -> PCIe TLPs through the root complex -> IOMMU translation -> NIC MAC -> PHY / SerDes -> the wire. A NIC engineer has to know which layer owns the byte, because a missed handoff surfaces only as "corrupt frame", "stuck queue", or "latency spike".

At the core, a store instruction retires after the pipeline produces an address and data. At about 3 GHz, one cycle is roughly 0.33 ns; the store can wait in the store buffer before draining into coherent cache. L1 is sub-nanosecond to about a nanosecond, L2 a few nanoseconds, shared last-level cache often tens of nanoseconds, and DRAM tens to roughly 100 ns or more. MESI-family coherence, historically MOESI-family on AMD systems, determines when another CPU or coherent DMA agent can observe the line.

The driver builds DMA-visible state: packet bytes in a mapped buffer and a TX descriptor with DMA address, length, flags, offload controls, and an ownership or valid bit. If an IOMMU is enabled, dma_map_*() gives the driver an I/O virtual address. On AMD-Vi, the IOMMU translates the device's IOVA or device virtual address, scoped by requester/domain, into a system physical address and permissions before the request reaches DRAM.

The first correctness handoff is release / publish ordering. Descriptor contents must precede the ownership flag. Otherwise another CPU or the NIC can observe "ready" while still reading stale address, length, or flags.

desc->addr = dma_addr;
desc->len = len;
desc->flags = flags;
dma_wmb();
desc->owner = DEVICE_OWN;

The second handoff is descriptor-before-doorbell ordering. Ring entries and packet buffers are normal coherent memory; the doorbell is an MMIO write into a PCIe BAR. The doorbell tells the NIC to fetch, so descriptor and buffer writes must be globally visible first. Linux uses dma_wmb() for DMA-memory writes before device ownership, and wmb() before writel() where the next operation is cache-incoherent MMIO.

dma_wmb();
wmb();
writel(tail, txq->doorbell);

The doorbell becomes a PCIe Memory Write Request TLP, normally a posted write, so the CPU does not wait for completion. A PCIe TLP has a 3 DW or 4 DW header, optional payload, and link-layer protection such as LCRC. The root complex accepts and routes it; PCIe is no longer core-scale. Root-complex or switch traversal is commonly hundreds of nanoseconds, and practical device round trips are often around 1 us before NIC queueing, firmware, or media effects.

After the doorbell, the NIC issues PCIe Memory Read requests for descriptors and packet data. These pass through the root complex and, if enabled, the IOMMU path; returned Completion TLPs carry bytes into the NIC. The MAC frames the data according to Ethernet configuration: preamble/SFD, addresses, EtherType or length, padding, FCS, and offloads. The PHY/SerDes serializes bits, applies encoding, scrambling, or FEC as appropriate, and drives the medium.

The third handoff is device completion-before-interrupt ordering. When the NIC reports work to the host by DMA-writing packet data or a completion entry and then raising MSI/MSI-X, the interrupt must not arrive before the data it announces. With default PCIe ordering attributes, posted writes from the same requester and traffic class remain ordered; Relaxed Ordering or ID-Based Ordering can weaken that producer-consumer guarantee. If this edge is wrong, the CPU handles an interrupt, reads the completion, and races stale memory.

This is why low-level driver work is whole-system work. The byte is successively owned by the CPU pipeline, coherent memory system, DMA API, PCIe fabric, IOMMU, NIC DMA engine, MAC, and PHY, and a bug at any layer can surface far away from where it was introduced.

2. How the send becomes executed instructions

It starts as a mov store. On a modern out-of-order core that store is fetched, decoded to µops, renamed, executed out of order, and retired in order— and even then it isn't in memory yet.

From C to a store instruction. A byte headed for the wire may begin as ordinary C:

*p = b;
desc->len = len;

For normal cacheable memory, an optimized x86-64 build will often emit a mov store such as movb %al,(%rdi) or movw %ax,0x10(%rdi). A transmit descriptor field store is the same kind of operation: an instruction that writes bytes to an address. But C is not a device protocol. The compiler may reorder, merge, or remove non-observable stores if C semantics allow it. Driver code uses volatile for true memory-mapped I/O registers, compiler barriers, atomics, and later CPU fences; here, assume one real store instruction survives.

Fetch is prediction-driven. The core fetches instruction bytes from the level-1 instruction cache, the L1i, typically 32 KiB on recent Zen and Intel Core designs. Fetch runs ahead under branch prediction. The Branch Target Buffer, or BTB, predicts taken-branch targets, while direction predictors guess conditional branches. Public Zen 4 data puts the L1 BTB around 1.5K entries and the L2 BTB around 7K; exact predictors are proprietary. In NIC fast paths, a mispredicted “ring full?” or “packet done?” branch can starve the backend before any descriptor store progresses.

Decode creates uops. x86 instructions are variable length, from 1 to 15 bytes, so the frontend predecodes boundaries and decodes instructions into internal micro-ops, or uops. A memory mov store becomes work for address generation and store-data machinery. Recent cores cache decoded work. Intel calls its decoded-uop cache the Decoded Stream Buffer, or DSB; AMD calls the analogous structure an op cache. Golden Cove is commonly described with about a 4K uop cache delivering up to 8 uops per cycle; Zen 4 increased the op cache to about 6.75K ops and can deliver about 9 macro-ops per cycle. Zen 5 changed the frontend again, with public AMD material describing simultaneous fetch/decode and op-cache paths. Macro-op fusion is a useful aside: compare-and-branch pairs can sometimes consume less frontend bandwidth.

Rename breaks false dependencies. Decoded uops enter register renaming. The Register Alias Table, or RAT, maps architectural registers such as rax and rdi onto a larger Physical Register File, or PRF. This removes false write-after-read and write-after-write dependencies: unrelated writes to architectural rax can use different physical registers. Public Zen 4 figures list roughly 224 integer physical registers and 192 floating/vector physical registers; Golden Cove-class sizes differ. Some instructions are handled here without execution: register moves may be eliminated by changing RAT mappings, and zeroing idioms such as xor eax,eax can create a known-zero result.

Allocate in order, execute out of order. Renamed uops allocate into the ReOrder Buffer, or ROB, in program order. The ROB tracks speculative work and provides precise retirement. Zen 4 has a 320 entry ROB; Golden Cove expanded to about 512; Zen 5 publicly grew several out-of-order structures, though exact counts vary by source. Uops also enter scheduler structures often called Reservation Stations, or RS, plus load/store queues. Allocation is in order: if the store cannot get a ROB or store-queue entry, younger work cannot pass it. Execution is out of order: once operands and resources are ready, independent younger uops may issue before older stalled uops.

Address generation is scheduled work. A store such as movb %al,(%rdi,%rcx,1) needs an effective virtual address: base plus index times scale plus displacement. An Address Generation Unit, or AGU, computes it. Golden Cove has 12 execution ports, including load, store-address, and store-data capability. Zen 4 has three AGU-capable memory pipes; Zen 5 public Hot Chips material describes four load/store pipes, up to four loads or two stores per cycle, and two store commits per cycle. Exact port maps vary, but the store address is still scheduled backend work.

Stores split into address and data. A store normally becomes a store-address uop, STA, and a store-data uop, STD. The STA records the computed address after translation; the STD supplies the bytes. Together they populate a store-buffer or store-queue entry. Zen 4’s store queue is about 64 entries; Golden Cove’s store buffer is commonly reported around 114 entries, with about 192 load-buffer entries. The crucial point for a driver engineer is that after STA and STD execute, the bytes are in the store buffer, not necessarily in the L1 data cache, DRAM, PCIe root complex, or NIC.

The Load-Store Queue speculates. The Load-Store Queue, or LSQ, tracks in-flight loads and stores. Loads may speculatively bypass older stores whose addresses are not known, using memory disambiguation to predict non-overlap. If an older store later resolves to the same address, the core detects a memory-order violation, squashes younger work, and replays. If a load depends on an older known store, store-to-load forwarding can provide bytes directly from the store buffer, though size and alignment mismatches can cause penalties.

Speculation can be squashed. The store may sit behind predicted branches, speculative memory ordering, speculative translation, and younger work issued out of order. A branch mispredict, memory-order violation, fault, or exception flushes younger uops, discards their unretired ROB and scheduler state, and restores the RAT from a checkpoint or retirement map. An unretired store must not leak into architectural memory state, so the store buffer distinguishes speculative entries from stores that are guaranteed to happen.

Retirement is not memory arrival. The ROB retires in program order. A store retires only when it is the oldest completed instruction and has no fault, making it architecturally visible. Only after retirement does it drain from the store buffer: to the L1 data cache for ordinary write-back memory, or to the uncacheable/write-combining path for memory-mapped I/O. Retirement therefore does not mean the descriptor is in DRAM or the doorbell has reached the NIC. A descriptor write can be retired while still buffered before becoming visible to coherent DMA; an MMIO doorbell can be retired while still queued toward PCIe. Correct NIC drivers choose memory types, DMA synchronization, compiler barriers, and CPU fences so that when the device observes the doorbell, the descriptor bytes it will fetch are already visible in the required order.

3. Virtual to physical: the MMU & TLB

The address the store computed is virtual. Before it touches memory the MMU must translate it — a TLB hit in ~1 cycle, or a full four-level page walk on a miss.

From AGU output to a physical address. The address-generation unit has produced a canonical x86-64 virtual address: in 4-level paging, bits 63:48 must all copy bit 47, giving 48 implemented virtual-address bits. Before that address can touch cache or memory, the MMU translates it through a radix tree of page tables rooted at CR3. In long mode with 4 KB pages, the split is exactly 9/9/9/9/12: bits 47:39 select a PML4 entry, bits 38:30 select a PDPT entry, bits 29:21 select a page-directory entry, bits 20:12 select a page-table entry, and bits 11:0 are the byte offset.

PML4 means page-map level 4. A PDPT is the page-directory-pointer table; a PDPTE is one entry in it. A PDE is a page-directory entry. A PTE is a page-table entry, the final descriptor for a 4 KB page. Each table is one 4 KB page containing 512 8-byte entries.

The walk. CR3 supplies the physical base address of the top-level table. With CR4.PCIDE=1, low 12 bits of CR3 are a PCID, a process-context identifier used to tag cached translations; the table base comes from the aligned high bits. Without PCID, CR3 names the current address-space root plus legacy caching controls. The hardware page walker reads the PML4E, checks it, uses its frame address plus the next index to find the PDPTE, then the PDE, then the PTE. The final physical address is the selected page-frame base plus the original page offset:

pa = pte.frame_base | (va & 0xfff);

Every step is permission-bearing. If an entry is absent, has illegal reserved bits, or denies access, the CPU raises a page fault before memory becomes visible.

Leaf bits driver engineers actually meet. In a 4 KB leaf PTE, bit 0 is P or present: if clear, the mapping is invalid and the OS can use the remaining bits for swap or metadata. Bit 1, R/W, allows writes when set. Bit 2, U/S, chooses user versus supervisor access. Bit 5, A, is the accessed bit, set by hardware when the translation is used. Bit 6, D, is the dirty bit, set by hardware on a write. Operating systems use A and D for replacement, copy-on-write, writeback, and dirty accounting; driver bugs that forget write permissions or race page-state transitions show up as faults or stale DMA assumptions.

Bit 63 is NX in AMD terminology and XD in Intel terminology, enabled by EFER.NXE; when set, instruction fetch is forbidden. That is the executable/non-executable boundary behind W^X kernel text, JIT hardening, and guard mappings. Bits 3 and 4, PWT and PCD, are page-level write-through and cache-disable controls. The PAT bit combines with PCD and PWT to form a 3-bit index into the page attribute table: for a 4 KB PTE the PAT bit is bit 7; for large-page leaves it is bit 12. This matters for NIC work. MMIO doorbells, BAR registers, and descriptor apertures need the intended memory type, often uncacheable or write-combining. Mapping a device register as write-back memory can merge, delay, speculate, or reorder accesses; mapping packet data too conservatively can destroy throughput.

Large pages stop early. A 2 MB page is selected when the PDE has PS=1; the walk stops at the page-directory level and bits 20:0 are the offset. A 1 GB page is selected when the PDPTE has PS=1; the walk stops at the PDPT level and bits 29:0 are the offset. Larger pages remove lower-level table reads and reduce TLB pressure. A translation lookaside buffer, or TLB, is the core's cache of virtual-to-physical translations. One 4 KB TLB entry covers 4 KB; one 2 MB entry covers 512 times as much memory; one 1 GB entry covers 262144 times as much. For kernel-bypass networking, DPDK-style packet pools, pinned RX/TX rings, and huge DMA buffers, hugepages mean fewer translation misses while walking descriptors and payloads at line rate. They also make pinning and IOMMU mapping coarser, so allocation, NUMA placement, and fragmentation matter.

The fast path is the TLB hierarchy. Modern x86 cores do not perform a four-load page walk on every access. They have split L1 instruction and data TLBs, iTLB and dTLB, plus a larger unified second-level TLB called the STLB, the second-level TLB. A TLB hit is on the order of a cycle and is normally overlapped with cache lookup. On a miss, a hardware page walker runs. Intel also documents paging-structure caches for upper-level PML4, PDPTE, and PDE information, so a TLB miss need not fetch all four paging-structure entries from memory. With cold TLBs and cold paging-structure caches, however, a 4 KB translation can require up to four dependent memory reads before the real data load can even be addressed. Those reads go through the cache hierarchy, but they are serialized by pointer dependency: the PML4E tells you where the PDPT is, and so on.

Invalidation is software's job. TLBs and paging-structure caches are not kept hardware-coherent with stores that modify page tables. After the kernel changes a present mapping, permission, or memory type, it must invalidate stale cached translations before relying on the new meaning. INVLPG invalidates translations for one linear address on the local logical processor. Reloading CR3 can invalidate broader state, with details affected by PCID. PCID lets translations from multiple address spaces coexist in the TLB, so a context switch need not throw away every cached translation. AMD uses the closely related term ASID, address-space identifier, for the same tagging idea in other contexts.

On SMP systems, invalidation becomes a shootdown: if another core might have cached the old mapping, the initiating CPU sends inter-processor interrupts and waits until targets acknowledge local invalidation. That wait is why map/unmap churn hurts more as core counts rise. For a driver, frequent pin, unpin, mmap, munmap, remapping, or changing cache attributes on hot paths can spend real time in shootdowns rather than moving packets. Speculation does not remove this rule; stale translations remain a software coherency problem.

LA57 aside. With 5-level paging, enabled by CR4.LA57, the CPU adds a PML5 level above PML4. The virtual address becomes 57 bits, with bits 63:57 copying bit 56, and the split becomes 9/9/9/9/9/12: one more upper-level index and one more worst-case table read.

CPU MMU is not device DMA. Everything above describes the CPU core translating addresses used by instruction fetches, loads, stores, atomics, and page walks. A NIC doing DMA does not look in the CPU TLB, does not use the CPU page walker, and does not consume the process virtual address that C code used. The device issues PCIe transactions using bus addresses, often called I/O virtual addresses or IOVAs. If translation is enabled for the device, those addresses are translated by the IOMMU using its own page tables and its own IOTLB, the I/O TLB. CPU mappings, PCID, INVLPG, and CPU hugepage TLB hits explain how the driver touches memory; IOMMU mappings and IOTLB behavior explain how the NIC reaches it on the bus.

4. Store buffer, caches, coherence & ordering

The retired store drains into L1d, where the cache hierarchy and the coherence protocol take over — and where x86-TSO's ordering rules (and your barriers) decide what the NIC will eventually see.

data = 42;
ready = 1;   // plain / relaxed

while (!ready)   // plain / relaxed
   ;
use(data);

Retired does not mean visible. After an x86 store instruction retires, the value is not necessarily in cache yet. It sits in the store buffer: a per-core queue of retired stores waiting for cache-line ownership and drain into L1d. Retirement commits the store for that hardware thread, but other cores and devices cannot observe it until the store buffer has made it coherent. A later same-core load to an address still covered by a live store-buffer entry can use store-to-load forwarding, receiving the new value before the store is globally visible.

Cache transfer is line based. x86-64 AMD and Intel systems use a 64 B cache line as the unit of coherence, fill, invalidation, writeback, and false sharing. Modern server cores typically have private L1d/L1i, private L2, and shared last-level cache (LLC, normally L3). Rough numbers: L1d is 32 KiB, about 4-5 cycles, roughly 1 ns; L1i is 32 KiB on many Intel server cores and 32-64 KiB on AMD Zen; L2 is 1-2 MiB on recent Xeons and 512 KiB-1 MiB on many AMD Zen parts, about 10-15 cycles or 3-5 ns; L3 is tens of MiB, often 30-60+ cycles or 10-25+ ns. An L1 miss checks L2, L3/LLC, then DRAM, commonly tens to over 100 ns.

The LLC participates in coherence. Older Intel client and pre-Skylake-SP server parts are inclusive: private-cache lines are represented in the LLC, so it can act as a snoop filter and LLC eviction may force private invalidation. Skylake-SP and later Xeon Scalable parts moved to a non-inclusive LLC on a mesh. AMD Zen-family L3 is non-inclusive and mostly victim-style: it may hold private-cache victims, but is not a superset of all L1/L2 contents. Coherence messages move over the on-die interconnect: Intel rings on older parts, Intel mesh on Xeon Scalable, and AMD Infinity Fabric between CCX/CCD complexes, I/O die, memory controllers, and coherent agents.

MESI names the basic states. MESI is Modified, Exclusive, Shared, Invalid. Modified means this cache has the only valid copy and is dirty. Exclusive means this cache has the only valid copy and is clean. Shared means the line is clean and may exist in multiple caches. Invalid means unusable. Intel is commonly described as MESIF, adding Forward: one shared clean copy is selected to respond. AMD is commonly described as MOESI, adding Owned: a dirty line may be shared, with the owner supplying data and later writing it back.

Stores require ownership. To modify one byte, the core needs writable ownership of the whole 64 B line: Exclusive or Modified in MESI terms. If the line is absent or only Shared, the core issues an RFO (Read For Ownership): a coherence transaction that obtains the line and invalidates other readable copies. Only then can the pending store merge into L1d and become visible.

Write-combining changes the path. WC is the x86 write-combining memory type, often used for memory-mapped I/O (MMIO) or PCIe BAR mappings. Stores to WC memory do not behave like write-back cached stores allocating normal L1d/LLC lines. They accumulate in write-combining/fill buffers, coalescing adjacent writes into bursts, often a full cache line, before flushing toward the uncore and PCIe. BAR writes and doorbells may use WC for throughput, but ordering needs explicit care.

x86-TSO is strong but not magic. Under x86 total store order (TSO) for ordinary write-back memory, stores remain ordered with stores, loads with loads, and a load from the same address as an older store gets the store-buffer value. The key relaxation is StoreLoad: a later load to a different address may execute before an earlier store has drained. Locked read-modify-write instructions using LOCK, and xchg with memory, are full barriers for ordinary memory ordering and participate in the global order of locked operations.

Fences and compiler barriers are different. sfence orders older stores before younger stores and matters for non-temporal or WC stores. lfence orders older loads before younger loads and is also a speculation barrier. mfence is a full load/store fence. A compiler barrier such as asm volatile("" ::: "memory") or Linux barrier() emits no CPU fence; it only stops compiler motion. On x86-TSO, a release store to write-back memory often needs only a compiler barrier because hardware preserves store-to-store order. On weakly ordered ISAs, release/acquire or fences may generate hardware barriers.

Publish means data before flag. In an SPSC ring, the producer fills descriptors, then publishes an index, flag, or ownership bit the consumer polls:

desc->addr = dma_addr;
desc->len = len;
/* release barrier */
ring->prod = next;

The required property is that descriptor fields are globally visible before ring->prod. On x86 write-back memory, store-to-store ordering provides the hardware part, while release semantics constrain the compiler and document the contract. On ARM, RISC-V, POWER, and many embedded SoCs, the same release/acquire contract may require a hardware barrier or release-store instruction.

DMA adds a device observer. On cache-coherent x86 servers, a PCIe NIC DMA read of host memory is coherent with CPU caches through the fabric/I/O path, such as Intel integrated I/O (IIO) or AMD's I/O die and Infinity Fabric. If a descriptor line is dirty in a CPU cache, the DMA read is satisfied coherently; the driver does not explicitly flush cache lines before DMA-to-device. Ordering still matters: before setting ownership/valid or ringing a doorbell, descriptor fields must be visible to the device. Linux dma_wmb() is the CPU-to-device write-ordering primitive. wmb() is a mandatory write barrier for broader write/I/O ordering. smp_wmb() is CPU-to-CPU SMP ordering and may compile away on UP; it is not the right API when the observer is a device.

Non-coherent DMA needs cache maintenance. Many ARM and embedded SoCs do not make DMA coherent with CPU caches. Software must clean/write back dirty CPU cache lines before DMA-to-device, and invalidate cache lines before consuming DMA-from-device data. A NIC/driver engineer cares because a missing clean can make the NIC read stale descriptors, a missing invalidate can make the CPU read old receive data, and a missing dma_wmb() can let the device observe valid before descriptor fields.

5. Handing it to the NIC: descriptor + doorbell

Software builds a TX descriptor pointing at the buffer's DMA address, issues a dma_wmb(), then rings the doorbell — an MMIO write that tells the NIC “there's work.”

Descriptor handoff. On the normal TX path, the driver hands ownership to the NIC by building a TX descriptor: a small hardware-defined record containing the buffer DMA address, byte length, and command/flags bits such as end-of-packet, checksum offload, VLAN insertion, timestamp request, or report-completion. The descriptor is placed in a TX descriptor ring that lives in DMA-coherent host memory, commonly allocated with dma_alloc_coherent(). Coherent does not remove ordering requirements.

dma_addr_t dma = dma_map_single(dev, skb->data, len, DMA_TO_DEVICE);
txd->addr = cpu_to_le64(dma);
txd->len_flags = cpu_to_le32(len | flags);
dma_wmb();
writel(next_tail, regs + TDT);

The descriptor address is not a C pointer. dma_map_single(struct device *dev, void *cpu_addr, size_t size, enum dma_data_direction dir) takes a CPU virtual address and returns a dma_addr_t, the device-visible DMA address. The CPU must not dereference a dma_addr_t; the NIC uses it in PCIe DMA read requests.

That address is not necessarily the CPU physical address. With an IOMMU, the NIC sees an IOVA, or I/O virtual address, which the IOMMU translates on the device side into host physical pages. Without an IOMMU it may resemble a bus/physical address, but portable drivers must not assume that. The DMA API lets Linux isolate devices, fit mappings into a device's DMA mask such as a 32-bit window, and use bounce buffers when the real buffer is not directly reachable. Writing virt_to_phys() into descriptors will fail under virtualization, SR-IOV, IOMMU isolation, or older address-limited hardware.

Ordering before ownership. The required sequence is: fill descriptor fields, execute dma_wmb(), then ring the doorbell. dma_wmb() is the Linux DMA write memory barrier: it orders CPU writes to coherent memory shared with a DMA-capable device, so descriptor contents are visible before the device consumes them. It may be lighter than a full wmb(); it is for shared DMA memory ordering, not for globally ordering every memory and MMIO operation.

A compiler barrier is insufficient because it only constrains code generation. It does not flush CPU write buffers, account for weakly ordered CPUs, or define visibility at the PCIe/DMA boundary. "x86 stores are ordered" is also not portable: the NIC is not another coherent CPU core. Also note the boundary: dma_wmb() itself does not order MMIO regions. Use the proper accessor, normally writel(), for the doorbell; avoid relaxed or raw stores unless device ordering is proved. The doorbell must come last because it tells the NIC that descriptors up to the new tail are ready. If the NIC sees the tail before address, length, or flags are visible, it can DMA the wrong bytes or consume an incomplete descriptor.

Ringing the doorbell. MMIO, memory-mapped I/O, is device register access through CPU loads/stores. A PCI BAR, or Base Address Register, describes a device address range that the OS maps as an __iomem region. A doorbell is a register write used as notification. Intel-style NICs give a concrete example: TDT, the Transmit Descriptor Tail register. Writing the new tail index to TDT tells the NIC that new TX descriptors are available. The NIC then DMA-fetches descriptors, DMA-reads frame data, performs offloads, and feeds the MAC transmit pipeline.

UC versus WC mappings. Control registers and doorbells are typically mapped UC, uncacheable. UC MMIO gives discrete, strongly ordered device accesses: one accessor produces one device transaction, without CPU caching or write combining. That is what a driver wants for "advance this queue", "ack this interrupt", or "set this enable bit".

PIO data windows are different. WC, write-combining, mappings let the CPU merge stores into larger burst writes and hold them temporarily before sending them to PCIe. That matters when software streams packet bytes or descriptor push data into a NIC aperture; without WC, narrow stores become many expensive MMIO writes. The cost is weaker ordering: WC writes can be combined, delayed, and reordered. Good drivers keep side-effect registers UC and use WC only for defined PIO/push regions, with explicit flush or barrier rules at the transition.

The doorbell is posted. A PCIe MMIO write to a NIC BAR is normally a posted Memory Write: fire-and-forget, with no PCIe Completion TLP and no automatic read-back. The CPU can retire the store before the NIC has acted on it. That is why a doorbell is low latency. A non-posted MMIO read has the opposite shape: the requester waits for a Completion with data, adding a PCIe round trip and often stalling the core. Read-back can prove arrival, but it is intentionally avoided on the TX hot path.

Latency shortcuts. Solarflare/AMD adapters provide paths that reduce the descriptor-DMA sequence. TX_PUSH is used in the ef_vi model when the TX ring is empty: the doorbell is rung and the packet buffer address is written in one shot, improving latency because the adapter need not first DMA-read the descriptor. The tradeoff is ring-state sensitivity and possible latency/throughput cost if software polls completions to decide whether push is available.

CTPIO, cut-through programmed I/O, goes further. Software streams a complete Ethernet frame across PCIe into a NIC PIO path, and in cut-through mode the adapter can begin transmitting on the wire before the whole frame has crossed PCIe. That removes descriptor fetch latency and overlaps PCIe ingress with MAC egress. Constraints matter: PIO resources are limited, fallback descriptors are required for failure cases, checksums must be completed in software on documented ef_vi paths, and cut-through support is adapter-, mode-, and link-speed-dependent. Conventional DMA optimizes CPU efficiency and batching; TX_PUSH and CTPIO spend CPU/MMIO bandwidth to shorten first-byte latency.

6. Across PCIe: the full mechanism

The doorbell store becomes a PCIe transaction. Here is the whole machine: the layered TLP / DLLP / PHY stack, posted vs non-posted vs completion, flow control, ordering, and the NIC bus-mastering host memory.

PCIe is a switched packet fabric, not a bus. Legacy parallel PCI was a shared electrical bus: many devices observed the same address, data, and control wires, arbitration decided who could drive them, and bus loading limited frequency and fanout. PCI Express replaced that with point-to-point serial links. Each link connects exactly two ports, and each lane is full duplex: one differential pair for transmit and one differential pair for receive. A NIC is not electrically sharing wires with a GPU or NVMe device; packets move hop by hop.

At the top is the root complex, integrated into the CPU or SoC. It bridges CPU cores, memory, interrupt controllers, and the PCIe fabric. Root ports point downstream. Switches fan out the fabric; to configuration software, switch ports look like PCI-to-PCI bridges, with secondary/subordinate bus numbers and routing windows. Endpoints are leaves. Your NIC is an endpoint: it implements configuration space, BARs, DMA requester logic, completion handling, and interrupt generation, but does not forward traffic for other devices. Topology matters because it determines bandwidth sharing, peer-to-peer reachability, reset scope, hotplug behavior, ACS/IOMMU policy, and bridge-window programming.

The transaction layer makes PCIe look like loads, stores, and messages. A TLP, or Transaction Layer Packet, is the packet format consumed and produced by the transaction layer. Its header identifies the operation and route: fields include format/type, traffic class, attributes, length in doublewords, requester ID, tag, byte enables, and an address for address-routed requests. Completions carry completer identity, completion status, byte count, lower address bits, the original requester ID, and the tag so the requester can match the response. A TLP may also carry data, as in Memory Write or Completion with Data.

Traffic has three classes. Posted requests do not require a completion; Memory Write and most Message requests are common examples. Non-posted requests require a completion; Memory Read, I/O, and Configuration requests are examples. Completion TLPs answer non-posted requests; CplD carries data, while Cpl carries no data. This is central to NIC performance: TX doorbells and DMA writes pipeline well, while descriptor and packet fetches consume tags, completion credits, host latency, and buffering.

The data link layer makes each hop reliable. PCIe reliability is per link, not end to end. The data link layer adds a sequence number and LCRC, the Link CRC, to each TLP. The receiver checks the LCRC and sequence, then returns ACK or NAK information using DLLPs. A DLLP, or Data Link Layer Packet, is a small control packet for acknowledgements, negative acknowledgements, power management, and flow-control updates. The transmitter keeps a replay buffer of unacknowledged TLPs and retransmits from the appropriate sequence point after corruption or loss.

The same layer runs credit-based flow control. A transmitter may send a TLP only when it holds enough receiver-advertised credits. Credits are separated into header credits and data credits, and also by posted, non-posted, and completion traffic. Header credits protect TLP-header queues; data credits protect payload buffers. A NIC can be throttled by root-port, switch, or endpoint buffer pressure before raw lane bandwidth is exhausted, so DMA engines need enough outstanding reads, completion buffering, and posted-write buffering to keep the link busy.

The physical layer moves symbols over lanes. Links train through the LTSSM, the Link Training and Status State Machine. States include Detect, Polling, Configuration, L0 for normal operation, and Recovery for retraining or equalization, plus power-management and reset states. Training discovers presence, lane polarity, lane reversal where supported, width, speed, equalization settings, and readiness for L0.

PCIe link width is written x1, x2, x4, x8, x16, meaning that many lanes in parallel. GT/s means gigatransfers per second per lane, not gigabytes per second and not aggregate bandwidth. Gen1 is 2.5 GT/s and Gen2 is 5 GT/s; both use 8b/10b, so only 8 of every 10 transmitted bits are useful before packet overhead. Gen3 is 8 GT/s, Gen4 is 16 GT/s, and Gen5 is 32 GT/s; these use 128b/130b, about 1.54% encoding overhead. Gen6 is 64 GT/s using PAM4 signaling, FLIT mode, and forward error correction, FEC.

Approximate one-direction math before TLP/DLLP overhead is: Gen3 lane 8e9 * 128 / 130 / 8, about 984.6 MB/s, so Gen3 x8 is about 7.88 GB/s; Gen4 lane 16e9 * 128 / 130 / 8, about 1.969 GB/s, so Gen4 x8 is about 15.75 GB/s. Real throughput is lower after TLP headers, LCRC, DLLPs, SKP ordered sets, completions, alignment, and small-packet effects. A 100 Gb/s port is roughly 12.5 GB/s before Ethernet overhead, so Gen3 x8 cannot sustain full-duplex line rate for high-end ports, while Gen4 x8 often can.

Configuration makes the NIC visible. Every PCI function has 256 bytes of legacy configuration space; PCIe extends this to 4096 bytes. Devices are addressed by BDF, meaning Bus, Device, Function. The conventional field widths are 8 bus bits, 5 device bits, and 3 function bits: up to 256 buses, 32 devices per bus, and 8 functions per device. Multifunction NICs, SR-IOV physical functions, and virtual functions all live in this naming world.

BAR means Base Address Register. During enumeration, firmware or the OS reads IDs and capabilities, sizes each BAR by probing it, allocates address space, and writes base addresses back into the BARs. A memory BAR maps a device register aperture into the CPU physical address map. A driver then uses an uncached or device-mapped virtual address for NIC registers; the access becomes PCIe MMIO, not normal cacheable memory. BAR sizing and bridge windows explain probe failures, lspci -vv memory windows, and how data-sheet register offsets become CPU physical addresses.

Now trace the byte. The driver has prepared a transmit descriptor in host memory, usually after DMA mapping the packet buffer and applying CPU memory barriers so descriptor contents are visible before notification. Then it rings the NIC doorbell:

writel(tail, txq->doorbell);

That CPU MMIO store targets an address inside the NIC BAR. The root complex converts it into an address-routed posted Memory Write TLP. The address falls within the NIC BAR window, so root and switch decoders route it downstream. Because it is posted, there is no completion TLP. The store is low latency and can be buffered, but the CPU does not synchronously learn that the endpoint accepted it or that an error occurred later. Drivers therefore need ordering barriers around descriptor publication, and sometimes readbacks as flushes.

The NIC’s DMA engine now bus-masters. To fetch the descriptor, it issues a tagged, non-posted Memory Read Request TLP upstream with requester ID, byte enables, length, attributes, and host physical address. The root complex, acting as completer for host memory, returns one or more CplD TLPs carrying descriptor bytes and echoing requester ID and tag. A read is a round trip: request propagation, memory service, and completion propagation. A write is fire-and-forget; a read is latency-bound unless the NIC has enough independent reads in flight.

Next the NIC fetches packet payload from the descriptor’s DMA address. The maximum Memory Read Request size is limited by Max_Read_Request_Size, or MRRS, in the PCIe Device Control register. The data returned in any one TLP is limited by Max_Payload_Size, or MPS, also configured there and constrained by every component on the path. If MRRS is 512 bytes and MPS is 256 bytes, one read can return as two CplD TLPs. Larger MRRS reduces request header overhead but can increase completion burstiness; larger MPS reduces per-byte TLP overhead but requires path support. These knobs affect PCIe efficiency, DMA design, ring prefetch, and small-buffer packet rate.

Ordering keeps the driver sane. In the normal producer-consumer model, without relaxed-ordering shortcuts, posted writes in the same direction are observed in order: a later posted write must not pass an earlier posted write with the same ordering requirements. Non-posted reads can also push or drain earlier posted writes because the completion cannot be returned until the ordering point is satisfied. When the NIC writes received packet data and descriptor status to host memory, those are posted Memory Write TLPs. If it then raises MSI-X, the interrupt is another posted Memory Write TLP. PCIe/MSI ordering means the interrupt-generating write cannot pass the earlier data writes, so by the time the CPU observes the interrupt, the data and status are visible.

MSI-X is Message Signaled Interrupts eXtended: a PCI capability that gives a function a table of message address/data pairs, typically one per queue or vector. The OS programs that table with host interrupt-controller addresses and data values. To interrupt, the NIC performs a posted Memory Write to the programmed address with the programmed data. There is no dedicated interrupt pin or sideband wire. This is elegant because queue completion, DMA writes, and notification ride the same ordered TLP fabric. The interrupt is the final ordered write in the device’s producer sequence.

7. The IOMMU checks and translates every DMA

The address in the NIC's TLPs is an IOVA. The IOMMU translates it to host physical and bounds-checks it — the reason a bus-mastering device can't scribble arbitrary memory.

What the NIC puts on PCIe: when a NIC DMA engine fetches a TX descriptor or writes an RX buffer, the address field in its PCIe Memory Read or Memory Write TLP is an IOVA, an I/O virtual address, not necessarily a host physical address. An IOMMU is the DMA-side memory-management unit between PCIe root-complex traffic and DRAM. It translates IOVA to HPA and checks permissions. Intel calls this VT-d, Virtualization Technology for Directed I/O; AMD calls it AMD-Vi; Arm exposes an SMMU.

The IOMMU first identifies the requester. In PCIe, the Requester ID is the function's BDF: 8 bits of bus, 5 bits of device, and 3 bits of function, for a 16-bit TLP requester-id field. BDF means Bus/Device/Function, shown as bb:dd.f. The IOMMU uses it to choose the translation context: the DMA address space this device may use. A domain is one set of IOVA to HPA mappings, possibly shared by trusted devices.

The vendor structures differ, but the shape is the same: requester identity selects a per-device context pointing at I/O page tables. In Intel VT-d, the requester ID selects a root-table entry by bus number; the root entry points to a context table; device/function bits select the context entry, which contains the domain identifier, address-width controls, translation type, and second-level page-table root. In AMD-Vi, the DeviceID derived from PCI BDF indexes a Device Table; the Device Table Entry, or DTE, contains validity, translation controls, interrupt controls, and the host page-table root pointer. In Arm SMMUv3, a StreamID selects a Stream Table Entry, or STE, which selects bypass, stage-1, stage-2, or nested translation state; with substreams, PASID is an SSID selecting per-process context.

This is not the CPU MMU reusing process page tables for ordinary DMA. The IOMMU walks its own page tables. Intel second-level translation commonly uses x86-like multi-level tables with 4 KB base pages and larger leaves such as 2 MB and 1 GB; hardware advertises adjusted guest address widths such as 39, 48, or 57 bits, corresponding to 3, 4, or 5 levels. Entries have present/read/write-style permission bits, so a DMA write to read-only memory faults if the address translates. With no valid mapping, the IOMMU reports a DMA-remapping fault and blocks the transaction.

The hot-path cache: an IOTLB, or I/O Translation Lookaside Buffer, is the IOMMU's cache of recent IOVA to HPA translations. On an IOTLB hit, translation is cheap. On a miss, the IOMMU performs a page walk: several dependent memory reads before NIC DMA can proceed. That miss path, plus invalidation ordering, is the main cost.

Mapping changes must be synchronized with those caches. When the kernel maps or unmaps an IOVA, it updates I/O page tables and issues invalidations: Intel queued invalidation descriptors, AMD-Vi command-buffer invalidation commands, or Arm SMMUv3 command-queue TLB invalidates plus completion synchronization. Unmapping every packet would add page-table work and serialized IOTLB invalidation. That is why DPDK and ixy allocate packet-buffer pools up front, pin them, map once, then recycle buffers.

Huge mappings matter because packet buffers are touched at high rate. A mempool backed by 2 MB hugepages, or 1 GB hugepages when configured, needs far fewer IOMMU leaves and IOTLB entries than 4 KB pages. One 2 MB entry covers 512 base pages; one 1 GB entry covers 262144 base pages. Fewer entries means fewer misses while the NIC streams descriptors and packets.

Why safety depends on this: a PCIe bus-master device can initiate memory transactions by itself. Without DMA remapping, a NIC firmware bug, corrupted descriptor ring, or malicious userspace driver could overwrite kernel memory. With an IOMMU, the device can touch only pages mapped into its domain with suitable permissions. That sandbox makes vfio-pci and VM passthrough defensible: the kernel pins guest or process memory, maps it into the device's IOMMU domain, gives out IOVA values, and hardware enforces the boundary. IOMMU groups expose which devices cannot be separated safely.

Kernel-bypass drivers live or die on this detail. DPDK and ixy typically allocate hugepage-backed mempools, pin pages so they cannot be swapped or migrated, map them through VFIO into the IOMMU domain, then write IOVA values into TX and RX descriptors. With vfio-pci and the IOMMU enabled, those descriptor addresses are translated DMA addresses and the NIC is confined. In no-IOMMU or identity/passthrough mode, the bus address is typically the host physical address, or a fixed DMA offset from it; faster, but unsafe. This is why virt_to_phys()-style descriptors fail once an IOMMU-backed DMA API is in charge.

The same IOMMU block usually also performs interrupt remapping for MSI and MSI-X. MSI is a device-generated memory write encoding an interrupt target and vector. Without interrupt remapping, a guest-assigned device could forge interrupt messages to CPUs or vectors it should not control. Interrupt-remapping tables validate and translate those messages, so safe passthrough needs DMA remapping and interrupt remapping.

ATS at a glance: PCIe ATS, Address Translation Services, lets an endpoint ask the translation agent for an address translation using an ATS Translation Request and receive an ATS Translation Completion. The device may cache the result in an ATC, Address Translation Cache, then issue DMA marked already translated, reducing root-complex IOTLB pressure. This improves performance but changes the trust model: the platform must trust the device to obey ATS permissions, honor invalidations, and not fabricate translated requests. PASID and shared virtual addressing tag DMA with a per-process address space instead of only a per-device domain.

8. Inside the NIC: DMA → MAC → PHY/SerDes → the wire

The NIC DMAs the payload in, the MAC frames it (preamble, addresses, EtherType, FCS), and the PHY line-codes, serializes, and drives it as a differential signal — bits are now physically on the wire.

DMA into the NIC. The NIC is now a PCIe bus master. Its TX DMA engine has fetched the descriptor, then issued Memory Read requests for the packet buffer; host memory returns the bytes in Completion-with-Data TLPs, CplD. The NIC deposits those payload bytes into on-chip TX buffering, usually a per-queue FIFO or SRAM, sometimes backed by a shared packet buffer. That buffer rate-matches two very different systems: PCIe delivers bursty completions affected by host memory, IOMMU, credits, and arbitration, while the Ethernet transmitter drains at a constant serial rate once a frame starts. It also gives the MAC enough data to assemble a frame. Store-and-forward waits for the whole frame; cut-through can begin with only enough header and payload margin, saving latency but making underrun and FCS timing harder.

The MAC builds the frame. MAC means Media Access Control: the layer-2 engine that frames packets, pads short payloads, enforces spacing, and computes the FCS. On transmit it emits:

• 7 bytes preamble, alternating 1/0 for receiver synchronization.
• 1 byte SFD, Start Frame Delimiter, value 0xD5, bit pattern 10101011.
• 6 bytes destination MAC address.
• 6 bytes source MAC address.
• 2 bytes EtherType-or-length.
• 46 to 1500 bytes payload for a standard untagged frame; jumbo frames use larger configured limits.
• 4 bytes FCS, Frame Check Sequence.

The minimum Ethernet frame from destination MAC through FCS is 64 bytes; if the payload is short, the MAC adds pad bytes. The preamble, SFD, and inter-frame gap are not counted in that 64 bytes. Between frames the MAC inserts an IFG, inter-frame gap, of 12 bytes or 96 bit times. NIC engineers care because small-packet throughput is dominated by preamble, minimum-frame padding, and IFG, not just payload bandwidth.

The FCS catches corruption. The FCS is a 32-bit Ethernet CRC using polynomial 0x04C11DB7. The MAC computes it over destination address, source address, EtherType-or-length, payload, and pad, but not over preamble or SFD, then appends it as the final 4 bytes. The receiver recomputes the CRC and normally drops frames that fail. A single flipped protected bit in transit becomes an FCS error counter increment, so rising FCS errors point first at cable, optics, equalization, SerDes, PMA/PMD, or MAC/PHY boundary problems. TCP checksum errors without FCS errors instead suggest offload metadata, capture location, or memory corruption.

Offloads can change what reaches the MAC. With TX checksum offload, the descriptor tells the NIC where the IP, TCP, or UDP checksum field is and what byte range to cover; the NIC fills it before final framing. With TSO, TCP Segmentation Offload, the host gives one large TCP buffer, for example about 64 KB subject to stack and device limits, plus a template header and MSS. The NIC slices it into MSS-sized segments, replicates and adjusts Ethernet/IP/TCP headers, updates lengths, sequence numbers, and applicable IDs/flags, computes each L3/L4 checksum, and then each Ethernet FCS. This saves descriptors, doorbells, cache misses, and per-packet CPU. The trap is that pre-offload captures may show giant packets or bogus checksums, while a bad checksum-start, offset, MSS, or header-template field can silently put corrupt checksums on the wire.

The PHY starts at the PCS. PHY means Physical Layer device. At its top is the PCS, Physical Coding Sublayer, which maps MAC data and control into a coded stream for clock recovery, transition density, DC behavior, block synchronization, and lane alignment. 1 Gigabit Ethernet 1000BASE-X uses 8b/10b: every 8 data bits become a 10-bit code-group, a 25% overhead relative to payload bits, so 1 Gb/s data uses a 1.25 GBd NRZ serial stream. 10G and faster BASE-R PHYs use 64b/66b: every 64 bits get a 2-bit sync header, producing 66 transmitted bits and 3.125% overhead. The 64 payload bits are scrambled by a self-synchronizing scrambler with polynomial x^58 + x^39 + 1; the sync header is not scrambled. Coding exists so the receiver can recover clock, find block boundaries, maintain usable DC/transition statistics, and align lanes.

Multi-lane links add deskew work. For higher rates the PCS distributes the encoded stream round-robin across multiple lanes and inserts alignment markers. The receiver uses those markers to identify lanes, compensate skew, reorder data, and rebuild the block stream. A historical 100GBASE-R example is 4 lanes of roughly 25G NRZ; modern 100G can be a single physical lane using PAM4, and 400G is commonly 4 lanes of 100G PAM4. Bad module capabilities, auto-negotiation, gearbox setup, FEC mode, polarity, or lane mapping can produce the interview-class failure where signal detect is good but PCS alignment never locks, or the link comes up with FEC corrections climbing.

The PMA and SerDes serialize it. Below the PCS is the PMA, Physical Medium Attachment. It contains the SerDes, Serializer/Deserializer. On TX it takes parallel coded words or blocks, performs gearbox or bit-muxing when PCS lane count or word width differs from physical lanes, and serializes each lane into a high-rate bitstream. On RX it recovers the clock, samples, deserializes, and passes aligned words upward.

The PMD drives the actual signal. PMD means Physical Medium Dependent: the electrical or optical front end. Over copper, including twisted pair, twinax, or backplane, it drives differential electrical signals and receives them after channel loss, reflection, crosstalk, and equalization. Over fiber, it modulates a laser or optical source and the far end detects light. NRZ, Non-Return-to-Zero, has two levels and carries 1 bit per symbol. PAM4, 4-level Pulse Amplitude Modulation, has four levels and carries 2 bits per symbol, doubling bits per baud versus NRZ but shrinking noise margin and typically requiring FEC. At this point the byte is voltage transitions or optical on/off or multi-level transitions; it has left the host entirely.

TX completion closes the loop. After transmission reaches the NIC-defined completion point, the NIC writes TX completion state to host memory as a posted PCIe Memory Write TLP, often a descriptor write-back or queue-head update. Subject to interrupt moderation, it raises MSI-X; that interrupt is itself a posted Memory Write TLP to the programmed message address/data. With normal PCIe ordering for writes from the same function, completion data lands before the interrupt is observed. Moderation or coalescing batches completions by count and/or timer, trading lower interrupt cost and higher throughput against higher latency and less precise per-packet timing.

9. The whole trip, timed

Every hop, in nanoseconds — and exactly which stages kernel-bypass, TX_PUSH, and CTPIO delete. This is the narration to have cold.

The end-to-end budget is a cost gradient. At 3 GHz, one CPU cycle is about 0.33 ns, so instruction fetch/decode/execute, register rename, ALU work, store-buffer enqueue, and a TLB hit live in handfuls of cycles. A hot L1 load is typically 4-5 cycles; L2 is low-teens; local LLC is tens of cycles, often 10-20 ns. A TLB miss may issue several cacheable page-table reads, growing from tens of cycles when the walk hits cache to hundreds when it reaches DRAM. DRAM is commonly 60-100 ns before queueing. Past descriptors, buffers, doorbells, PCIe TLPs, MAC frames, SerDes symbols, and line transitions, the budget is tens to hundreds of nanoseconds. PCIe read round trips are commonly 400-800 ns, approaching or exceeding 1 us with topology or congestion.

The asymmetry matters. A CPU readl() from MMIO becomes a PCIe non-posted Memory Read: request TLP out, completion TLP back, and no value until the round trip finishes. The same applies when the NIC fetches a TX descriptor or packet payload: DMA reads are non-posted Memory Reads waiting for completions. Posted writes are different: the TX doorbell writel(), RX DMA packet writes, completion writes, and MSI/MSI-X interrupt messages are posted Memory Writes. They must drain, but the issuer does not wait for a completion. This is why fast paths avoid MMIO read-backs and descriptor read-backs.

The normal path is therefore: CPU publishes bytes and descriptors through virtual addresses; the MMU translates virtual to physical through the TLB or page walk; caches and coherence make dirty lines visible; the driver writes DMA addresses that are IOVAs when IOMMU/AMD-Vi is active; then it rings the NIC BAR doorbell with MMIO. The NIC translates IOVA to physical, fetches descriptors and TX data unless pushed, performs MAC framing and FCS, serializes through PHY/SerDes, and puts bits on the medium. A 64 B frame is only 51.2 ns at 10 Gb/s and 20.48 ns at 25 Gb/s; software, cache misses, DMA reads, PCIe completions, and wakeups dominate.

• Onload / ef_vi: maps the Solarflare/Xilinx/AMD virtual interface, queues, and event rings into the app, deleting the syscall, kernel TCP/IP traversal, socket-buffer copy, and interrupt-driven wakeup/context-switch path.
• DPDK: binds the NIC to vfio-pci or UIO and uses a poll-mode driver, deleting kernel stack and datapath interrupts.
• SO_BUSY_POLL / NAPI busy-poll: spins briefly in the socket or NAPI path, deleting interrupt latency and sleep/wakeup/context-switch time.
• TX_PUSH: sends the descriptor with the doorbell path, so the NIC need not perform the first descriptor DMA read before it knows what to transmit.
• CTPIO / cut-through PIO: writes the frame itself into the NIC through PIO. For small TX frames, cut-through mode can start transmitting while bytes are still arriving over PCIe, removing the host-memory DMA-read round trip from the critical path.

The same barriers keep reappearing. When publishing a descriptor, use smp_store_release() for the ownership or valid field, or dma_wmb() before setting it, so address, length, flags, and buffer contents become visible before the NIC sees ownership. Without it, the device can fetch a stale descriptor. Before ringing the doorbell, use dma_wmb() for DMA-visible memory and wmb() before writel() where coherent-memory-to-MMIO ordering is required, so descriptor and buffer are visible before the NIC is told to go. Without it, the doorbell can outrun the data. On completion, PCIe posted-write ordering for the same requester and ordering attributes keeps DMA packet/completion writes ahead of the MSI-X interrupt write; if relaxed or broken, the CPU can observe the interrupt before its data.

That is the interview model. You should be able to point to the ring buffer, ownership bit, doorbell register, virtual-to-physical translation, IOVA-to-physical IOMMU translation, and barrier between producers and consumers. A missing barrier or stray non-posted read at any layer becomes a corrupt frame, stuck queue, or latency spike. Debugging it requires holding the whole chain at once: core, cache, DRAM, DMA, PCIe ordering, NIC queues, MAC/PHY timing, and the wire.