src/memory.h

struct pkt_buf and struct mempool — the 64-byte buffer header.

Walkthrough, interview notes & deep dive

The memory management subsystem. memory.h defines the primitives for DMA-capable memory and packet buffer pools, which are fundamental to high-performance user-space networking. The header abstracts the complexities of huge pages—noted by HUGE_PAGE_BITS and HUGE_PAGE_SIZE at 2MiB—and physical address translation into a small set of structures: struct dma_memory, struct mempool, and struct pkt_buf. The struct dma_memory is a simple utility container that pairs a virtual address virt with its corresponding physical address phy, ensuring a region is reachable by both the CPU and the NIC hardware.

The mempool structure. To eliminate the overhead of standard malloc during packet processing, ixy uses a pre-allocated pool of buffers. The struct mempool manages these through a virtual base_addr, a fixed buf_size, and a count of num_entries. Allocation is a stack: free_stack_top indexes into the free_stack[] flexible array, which stores entry IDs rather than full pointers. As the source comment notes, the address of a buffer is base_addr + entry_id * buf_size. Popping or pushing an ID gives O(1) alloc and free; swapping the stack for a lock-free queue is all it would take to make the pool thread-safe.

struct mempool {
	void* base_addr;
	uint32_t buf_size;
	uint32_t num_entries;
	uint32_t free_stack_top;
	uint32_t free_stack[];
};

The pkt_buf header. Every packet is managed via a 64-byte struct pkt_buf header, asserted at compile time with static_assert(sizeof(struct pkt_buf) == 64). This fits a single CPU cache line. It stores buf_addr_phy, the exact physical address the NIC's DMA engine reads or writes; embedding it here lets the driver populate a DMA descriptor with no page-table lookup. The mempool back-pointer and mempool_idx let pkt_buf_free return the buffer to the right pool in constant time, size records the packet length, and head_room[SIZE_PKT_BUF_HEADROOM] (40 bytes) pads so that the flexible data[] member lands at offset 64, aligned to a cache line.

struct pkt_buf {
	uintptr_t buf_addr_phy;
	struct mempool* mempool;
	uint32_t mempool_idx;
	uint32_t size;
	uint8_t head_room[SIZE_PKT_BUF_HEADROOM];
	uint8_t data[] __attribute__((aligned(64)));
};

struct pkt_buf: phys addr, mempool ptr, size, then the data[] payload

The prototypes. The header exports the public API for the subsystem. memory_allocate_dma(size, require_contiguous) performs the raw allocation of pinned, DMA-able (optionally physically contiguous) memory. memory_allocate_mempool(num_entries, entry_size) layers the buffer-pool logic on top. For the hot path, pkt_buf_alloc_batch retrieves many buffers at once to refill RX rings, pkt_buf_alloc fetches a single buffer, and pkt_buf_free returns one. Two helpers, get_vfio_container and set_vfio_container, expose the global VFIO container fd used by the IOMMU path.

Interview angles

Why store buf_addr_phy in every buffer? So the driver can hand the physical (bus) address straight to a NIC DMA descriptor in O(1), with no virtual-to-physical translation on the per-packet hot path.
Why is data[] a flexible array member at the end? The payload is then contiguous with its header, so metadata and the start of packet data fetch together; the buffer is one allocation with no second pointer indirection.
What is head_room for? It pads the 24-byte fields out so data[] begins at offset 64, on a fresh cache line (and __attribute__((aligned(64))) enforces it), avoiding split/unaligned accesses and leaving room to prepend encapsulation headers without a copy.

Going deeper

struct pkt_buf {
	uintptr_t buf_addr_phy;
	struct mempool* mempool;
	uint32_t mempool_idx;
	uint32_t size;
	uint8_t head_room[SIZE_PKT_BUF_HEADROOM];
	uint8_t data[] __attribute__((aligned(64)));
};

The header layout is a masterclass in cache-line alignment. On 64-bit systems, the first four fields occupy exactly 24 bytes (8+8+4+4). By defining SIZE_PKT_BUF_HEADROOM as 40, the data[] flexible array member is pushed exactly to offset 64. The aligned(64) attribute ensures that even if the pkt_buf struct was shorter, the packet payload—which is the target of DMA transfers—starts on a new cache line. This prevents "false sharing" where the CPU updating packet metadata (like size) would invalidate the cache line containing the payload being processed by the NIC or another core.

static_assert(sizeof(struct pkt_buf) == 64, "pkt_buf too large");
static_assert(offsetof(struct pkt_buf, data) == 64, "data at unexpected position");
static_assert(offsetof(struct pkt_buf, head_room) + SIZE_PKT_BUF_HEADROOM == offsetof(struct pkt_buf, data), "head room not immediately before data");

These compile-time checks enforce the driver's structural invariants. If an engineer adds a new field for timestamping or VLAN offloading, the first assertion will fail, forcing a reduction in SIZE_PKT_BUF_HEADROOM to maintain the 64-byte boundary. The second assertion ensures that data[] doesn't "float" due to compiler-inserted padding between the fixed-size members and the flexible array. The third confirms the headroom is contiguous with the data, allowing protocol headers (like VXLAN or IPsec) to be prepended by simply decrementing the data pointer into the headroom space.

Harder interview questions

Q: Why use a stack of IDs (uint32_t) in the mempool instead of a stack of virtual pointers? A: Memory efficiency and translation. On 64-bit systems, uint32_t saves 4 bytes per entry compared to a void*. More importantly, the ID represents a fixed offset from base_addr. Since the mempool is allocated as a contiguous DMA region, the virtual-to-physical mapping is a simple linear calculation, making it easier to verify that the entire pool resides within the allocated hugepages.

Q: What are the performance implications of using a LIFO stack for buffer allocation? A: A LIFO (Last-In, First-Out) approach maximizes temporal cache locality. The buffer most recently returned to the pool (the "hottest" in the CPU cache) is the first to be re-allocated. In high-pps forwarding, this ensures that the pkt_buf header and the first few bytes of the payload are likely already in L1 or L2 cache when the next packet arrives.

Q: How does the driver handle memory safety if pkt_buf_free is called with a corrupt pointer? A: It doesn't, by design. This is a "u-driver" (user-space driver) where performance is prioritized. The mempool pointer is stored in the header to allow O(1) freeing, but if the pointer is garbage, the driver will perform an invalid memory access. A senior engineer would suggest adding a "magic number" or cookie to the header in debug builds to detect such corruptions.

Q: Why is HUGE_PAGE_SIZE fixed at 2MiB via HUGE_PAGE_BITS 21? A: Standard x86_64 hugepages are 2MiB. Using hugepages reduces TLB (Translation Lookaside Buffer) pressure. With 4KB pages, a 2GB mempool requires 524,288 TLB entries; with 2MiB pages, it requires only 1,024. This drastically reduces TLB misses during high-speed DMA operations where the NIC is frequently accessing different parts of the mempool.

Gotchas

Flexible Array Allocation: The struct mempool ends with free_stack[]. A common error is using sizeof(struct mempool) for allocation, which ignores the stack memory. One must allocate sizeof(struct mempool) + (num_entries * sizeof(uint32_t)).
Physical Contiguity: memory_allocate_dma takes a require_contiguous flag. If a mempool spans multiple 2MiB hugepages, they must be physically contiguous for the hardware's DMA engine to treat the entire pool as a single block. If they aren't, the simple offset math for physical addresses might point to unrelated system memory.

From ixy to a production driver

Kernel-owned packet memory In a Linux NIC driver, received packets usually become struct sk_buff objects. The sk_buff is metadata; bytes live in head buffers or page fragments. Modern drivers avoid hot-path allocation by recycling pages through page_pool, then attaching data with build_skb. The Intel ixgbe path for 82599-class adapters shows this directly: Rx pages can be reused by handing the other half of a page back to the ring, with dma_sync_single_range_for_cpu before CPU access. ixy collapses this into fixed-size pkt_buf objects from one mempool, teaching ownership but not general network-stack design.

DMA addresses are not just physical addresses Production kernel drivers allocate coherent descriptor memory with dma_alloc_coherent or map streaming packet buffers with dma_map_single/page variants, receiving a dma_addr_t that may be an IOMMU IOVA, SWIOTLB bounce address, or bus address. The DMA API also defines cache coherency, including sync_for_cpu and sync_for_device on non-coherent systems. ixy instead stores buf_addr_phy and uses pre-pinned 2 MiB hugepages so the NIC can DMA into stable memory. On x86 cache-coherent machines, with hugepage memory treated as DMA-stable, the example can show driver mechanics without every architecture's DMA rules.

The DPDK version of the same idea DPDK is closer to ixy's user-space model, but richer. rte_mempool supports bulk allocation and per-lcore caches to avoid contention. struct rte_mbuf is the grown-up cousin of struct pkt_buf: it has buffer addresses, pool ownership, packet metadata, RTE_PKTMBUF_HEADROOM of 128 bytes by default, refcounts for cloning, nb_segs, and m->next chaining for multi-segment packets. EAL supplies hugepages, giving applications pinned, DMA-capable memory without each driver reinventing the allocator.

What ixy leaves out on purpose The single free_stack is not thread-safe, has no per-core cache, and has no per-NUMA-node placement. pkt_buf has no refcount, clone support, external buffer attachment, or scatter/jumbo chain; a bad free can corrupt the pool because there is no magic cookie. These omissions keep ownership visible. In an interview, expected production additions are synchronization or per-core pools, NUMA-aware allocation, refcounting, multi-segment buffers, validation, and proper DMA/IOMMU integration.

Isolation boundary With VFIO, ixy can use the global container fd to map DMA through an IOMMU, giving user-space drivers a controlled DMA aperture. In UIO-style/raw-physical mode, the NIC can DMA to addresses the program gives it, which is fast to explain but unsafe. Production designs use the kernel DMA API or VFIO/IOMMU so device access is pinned, translated, and isolated.

Sources

Source

filesrc/memory.h

#ifndef IXY_MEMORY_H
#define IXY_MEMORY_H

#include <assert.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <unistd.h>

struct ixy_device;

#define HUGE_PAGE_BITS 21
#define HUGE_PAGE_SIZE (1 << HUGE_PAGE_BITS) // 2_097_152 = 2MiB
#define SIZE_PKT_BUF_HEADROOM 40

struct pkt_buf {
	// physical address to pass a buffer to a nic
	uintptr_t buf_addr_phy;
	struct mempool* mempool;
	uint32_t mempool_idx;
	uint32_t size;
	uint8_t head_room[SIZE_PKT_BUF_HEADROOM];
	uint8_t data[] __attribute__((aligned(64)));
};

static_assert(sizeof(struct pkt_buf) == 64, "pkt_buf too large");
static_assert(offsetof(struct pkt_buf, data) == 64, "data at unexpected position");
static_assert(offsetof(struct pkt_buf, head_room) + SIZE_PKT_BUF_HEADROOM == offsetof(struct pkt_buf, data), "head room not immediately before data");

// everything here contains virtual addresses, the mapping to physical addresses are in the pkt_buf
struct mempool {
	void* base_addr;
	uint32_t buf_size;
	uint32_t num_entries;
	// memory is managed via a simple stack
	// replacing this with a lock-free queue (or stack) makes this thread-safe
	uint32_t free_stack_top;
	// the stack contains the entry id, i.e., base_addr + entry_id * buf_size is the address of the buf
	uint32_t free_stack[];
};

struct dma_memory {
	void* virt;
	uintptr_t phy;
};

struct dma_memory memory_allocate_dma(size_t size, bool require_contiguous);

struct mempool* memory_allocate_mempool(uint32_t num_entries, uint32_t entry_size);
uint32_t pkt_buf_alloc_batch(struct mempool* mempool, struct pkt_buf* bufs[], uint32_t num_bufs);
struct pkt_buf* pkt_buf_alloc(struct mempool* mempool);
void pkt_buf_free(struct pkt_buf* buf);

// reads the global VFIO container
int get_vfio_container();

// globally sets the VFIO container
void set_vfio_container(int fd);

#endif //IXY_MEMORY_H