src/driver/device.h

Walkthrough, interview notes & deep dive

The Role of device.h serves as the primary abstraction layer for the ixy driver. In a low-level C environment, this file defines the "interface" that any supported hardware must implement. It separates the generic device management logic from the vendor-specific register manipulations found in the driver implementation. By using this header, applications can interact with a struct ixy_device without needing to know whether the underlying hardware is an Intel 82599 (ixgbe) or a virtualized VirtIO device.

The Hardware Abstraction via struct ixy_device is implemented using a classic C design pattern: the manual vtable. Instead of language-level objects, ixy uses a structure containing metadata and function pointers. Key members include pci_addr and driver_name for identification, alongside num_rx_queues and num_tx_queues for resource management. The core of the datapath is defined by function pointers like rx_batch and tx_batch.

struct ixy_device {
	const char* pci_addr;
	const char* driver_name;
	uint16_t num_rx_queues;
	uint16_t num_tx_queues;
	uint32_t (*rx_batch) (struct ixy_device* dev, uint16_t queue_id, struct pkt_buf* bufs[], uint32_t num_bufs);
	uint32_t (*tx_batch) (struct ixy_device* dev, uint16_t queue_id, struct pkt_buf* bufs[], uint32_t num_bufs);
    // ... other vtable entries
};

Function Pointer Dispatch allows for high-performance polymorphism. When an application calls ixy_rx_batch, it is actually executing a static inline wrapper that forwards the call to dev->rx_batch(...). This minimizes the overhead of the abstraction while allowing the driver to swap implementations at runtime. Other control-plane operations like set_promisc, get_link_speed, and read_stats follow the same pattern, ensuring the user-facing API remains consistent regardless of the NIC model.

MMIO Mechanism and Register Access are handled through the uint8_t* addr (often representing the Base Address Register 0, or BAR0). In high-speed networking, the CPU interacts with the NIC by reading and writing to specific memory addresses that are mapped to hardware registers. This header provides the critical primitives for this interaction: get_reg32 and set_reg32.

The Importance of Volatile and Memory Barriers cannot be overstated in this context. The helpers use volatile uint32_t* casting to tell the compiler that the value at the register address can change outside the program's control (i.e., by the hardware). This prevents the compiler from optimizing out "redundant" reads or caching register values in CPU registers. Furthermore, the use of __asm__ volatile ("" : : : "memory") acts as a compiler barrier. This ensures that the compiler does not reorder memory instructions around the register access, which is vital when the order of operations (like writing a tail pointer after preparing descriptors) dictates hardware behavior.

Register Helper Functions simplify complex hardware interactions into readable logic. set_flags32 and clear_flags32 allow for atomic-like bit manipulation (though not CPU-atomic in the multi-core sense) by reading, masking, and writing back. More importantly, wait_set_reg32 and wait_clear_reg32 implement polling loops with usleep and repeated barrier checks. These are used during device initialization or reset sequences, where the driver must wait for the hardware to acknowledge a state change or clear a "busy" bit in a status register.

Interview angles

• What is the purpose of the volatile keyword in the get_reg32 function? It prevents the compiler from optimizing away repeated reads to the same address, ensuring the CPU actually fetches the latest value from the hardware register every time.
• Why does ixy use a struct of function pointers instead of a switch statement in the datapath? Function pointers provide a cleaner abstraction for multiple driver support (polymorphism) and avoid the branch misprediction penalty of a large switch statement in the hot path.
• Explain the significance of __asm__ volatile ("" : : : "memory"). It is a compiler barrier that prevents the compiler from reordering memory accesses across the barrier, ensuring that hardware side effects occur in the exact order specified by the C code.
• How does container_of work and why is it useful here? It calculates the starting address of a parent structure given a pointer to one of its members. This allows the generic ixy_device to be embedded inside a driver-specific struct (like ixgbe_device) while still allowing the driver to recover the full context.

Going deeper

#define container_of(ptr, type, member) ({\
	const typeof(((type*)0)->member)* __mptr = (ptr);\
	(type*)((char*)__mptr - offsetof(type, member));\
})

Type Safety in Statement Expressions. While the basic goal is pointer arithmetic, this GCC extension uses typeof to create a temporary pointer __mptr. This is a crucial compile-time check: if the type of ptr does not match the type of member, the compiler will issue a warning. Without this intermediate assignment, a raw cast would silently accept incompatible pointers, leading to catastrophic runtime memory corruption when the offset is applied.

while (cur = *((volatile uint32_t*) (addr + reg)), (cur & mask) != 0) {
	debug("waiting for flags 0x%08X...", mask, reg, cur);
	usleep(10000);
	__asm__ volatile ("" : : : "memory");
}

The Comma Operator and Loop Re-evaluation. The wait_clear_reg32 function uses the comma operator inside the while condition to perform an assignment and a logical check in a single expression. Because cur is assigned inside the condition, it is refreshed on every iteration. The __asm__ barrier inside the loop body is not redundant; it prevents the compiler from assuming that because cur didn't change in the *previous* iteration's code, it can skip the load in the *next* iteration.

while ((num_sent += ixy_tx_batch(dev, queue_id, bufs + num_sent, num_bufs - num_sent)) != num_bufs) {
	// busy wait
}

Batch Partial Success Handling. The ixy_tx_batch_busy_wait function handles "short writes" where the NIC ring buffer is full. It uses pointer arithmetic (bufs + num_sent) to advance the window of packets being sent. This assumes the driver implementation of tx_batch is idempotent regarding the pkt_buf state—if a packet isn't accepted, it must be left in a state where it can be retried immediately without modification.

Harder interview questions

• Why is the empty __asm__ barrier sufficient for x86 but dangerous for ARM-based SmartNICs? x86 follows Total Store Order (TSO), which prevents most hardware reordering. ARM is weakly ordered; an empty compiler barrier only stops the compiler, not the CPU. On ARM, you would need a dmb (Data Memory Barrier) to ensure a register write (like a doorbell) is visible to the NIC hardware after the DMA descriptors are written.
• What is the performance cost of the PCI IO-port helpers like read_io32? Unlike MMIO which uses standard load/store instructions, these helpers use pread and pwrite on a file descriptor. This triggers a context switch to the kernel for every single register access. In a high-speed data path, this would be a massive bottleneck compared to the direct memory mapping used in get_reg32.
• Why does wait_clear_reg32 use usleep(10000) instead of a simple cpu_relax() or pause instruction? The 10000 microseconds (10ms) is extremely slow for a NIC. This suggests the function is intended only for infrequent configuration/initialization (like a global reset). In a high-performance polling loop, 10ms would drop millions of packets; a senior engineer would replace this with a bounded cycle-count loop for lower latency.

Gotchas

• Infinite Spin Hazard: The wait_*_reg32 functions have no timeout mechanism. If the hardware hangs, the PCIe link flaps, or a "surprise removal" occurs, the driver will hang the entire thread indefinitely with no way to recover.
• Posted Write Invisibility: PCIe writes are "posted" (fire-and-forget). If you write a configuration register and immediately depend on that change in the next line of code, the hardware might not have processed it yet. A "read-back" (calling get_reg32 on the same register) is usually required to flush the write buffer.
• Alignment Performance: The mac_address struct is __attribute__((__packed__)). While this ensures it is exactly 6 bytes for hardware compatibility, it can cause the compiler to generate inefficient byte-by-byte move instructions instead of a single 64-bit load/store, as the address may no longer be 8-byte aligned.

From ixy to a production driver

ixy's struct ixy_device is the same architectural trick as a production NIC stack, with the scaffolding stripped away: generic code calls a small operation table, while each real driver embeds that generic header and recovers its concrete type with container_of. Linux does this more broadly through struct net_device: struct net_device_ops holds lifecycle and datapath hooks such as ndo_open, ndo_stop, and ndo_start_xmit, while struct ethtool_ops carries management hooks for link settings, statistics, and feature reporting. DPDK's PMDs similarly plug callbacks into struct eth_dev_ops under struct rte_eth_dev; for speed, DPDK keeps burst callbacks directly in the device structure, and rte_eth_rx_burst / rte_eth_tx_burst dispatch through them. ixy's rx_batch, tx_batch, read_stats, set_promisc, get_link_speed, get_mac_addr, and set_mac_addr are the teaching-sized version.

The register layer is where the simplification matters most. A Linux PCI driver maps a BAR with ioremap or pci_iomap and then uses readl() / writel(), because those helpers encode MMIO ordering and endian rules. DPDK does the userspace version: rte_read32() does a relaxed volatile load followed by rte_io_rmb(), and rte_write32() does rte_io_wmb() before the relaxed store. ixy's get_reg32() and set_reg32() use a volatile 32-bit load/store plus __asm__ volatile("" : : : "memory"). That empty asm is only a compiler barrier: it emits no CPU fence. On x86 this is usually acceptable for normal MMIO ordering because TSO is strong enough, mirroring DPDK's cheap x86 behavior. On weakly ordered ARM it would be wrong without real dmb/dsb barriers.

The wait_set_reg32() and wait_clear_reg32() loops show the boundary too. Real ixgbe-style initialization touches 82599 registers such as CTRL and STATUS, including reset bits, but it bounds waits and has recovery paths for failed reset, EEH, or surprise unplug. ixy spins forever because the pedagogical path is "program the NIC or fail loudly."

Interviewers usually want you to name what is missing and why it is missing:

• No locking or per-queue spinlocks for multi-threaded control paths.
• No timeout/error recovery around register waits.
• No full multiqueue/RSS control plane beyond passing a queue id.
• No interrupt coalescing, MSI-X, or NAPI-style polling in this layer.
• No checksum, TSO, RSC/LRO, VLAN, or timestamp offload negotiation.
• No broad ethtool stats, hotplug, power management, or suspend/resume story.
• The PCI IO-port helpers use pread()/pwrite(), so every access is a syscall; real fast paths prefer mapped MMIO, with config exposed through PCI support or sysfs.

Sources

• Linux networking netdevices documentation
• Linux bus-independent device access documentation
• DPDK rte_ethdev API documentation
• Intel 82599 10 Gigabit Ethernet Controller Datasheet

Source

#ifndef IXY_DEVICE_H
#define IXY_DEVICE_H

#include <stdint.h>
#include <unistd.h>

#include "log.h"
#include "memory.h"
#include "../interrupts.h"

#define MAX_QUEUES 64

// Forward declare struct to prevent cyclic include with stats.h
struct device_stats;

struct __attribute__((__packed__)) mac_address {
	uint8_t	addr[6];
};

/**
 * container_of - cast a member of a structure out to the containing structure
 * Adapted from the Linux kernel.
 * This allows us to expose the same struct for all drivers to the user's
 * application and cast it to a driver-specific struct in the driver.
 * A simple cast would be sufficient if we always store it at the same offset.
 * This macro looks more complicated than it is, a good explanation can be
 * found at http://www.kroah.com/log/linux/container_of.html
 * @ptr:	the pointer to the member.
 * @type:	the type of the container struct this is embedded in.
 * @member:	the name of the member within the struct.
 *
 */
#define container_of(ptr, type, member) ({\
	const typeof(((type*)0)->member)* __mptr = (ptr);\
	(type*)((char*)__mptr - offsetof(type, member));\
})

struct ixy_device {
	const char* pci_addr;
	const char* driver_name;
	uint16_t num_rx_queues;
	uint16_t num_tx_queues;
	uint32_t (*rx_batch) (struct ixy_device* dev, uint16_t queue_id, struct pkt_buf* bufs[], uint32_t num_bufs);
	uint32_t (*tx_batch) (struct ixy_device* dev, uint16_t queue_id, struct pkt_buf* bufs[], uint32_t num_bufs);
	void (*read_stats) (struct ixy_device* dev, struct device_stats* stats);
	void (*set_promisc) (struct ixy_device* dev, bool enabled);
	uint32_t (*get_link_speed) (const struct ixy_device* dev);
	struct mac_address (*get_mac_addr) (const struct ixy_device* dev);
	void (*set_mac_addr) (struct ixy_device* dev, struct mac_address mac);
	bool vfio;
	int vfio_fd; // device fd
	struct interrupts interrupts;
};

struct ixy_device* ixy_init(const char* pci_addr, uint16_t rx_queues, uint16_t tx_queues, int interrupt_timeout);

// Public stubs that forward the calls to the driver-specific implementations
static inline uint32_t ixy_rx_batch(struct ixy_device* dev, uint16_t queue_id, struct pkt_buf* bufs[], uint32_t num_bufs) {
	return dev->rx_batch(dev, queue_id, bufs, num_bufs);
}

static inline uint32_t ixy_tx_batch(struct ixy_device* dev, uint16_t queue_id, struct pkt_buf* bufs[], uint32_t num_bufs) {
	return dev->tx_batch(dev, queue_id, bufs, num_bufs);
}

static inline void ixy_read_stats(struct ixy_device* dev, struct device_stats* stats) {
	dev->read_stats(dev, stats);
}

static inline void ixy_set_promisc(struct ixy_device* dev, bool enabled) {
	dev->set_promisc(dev, enabled);
}

static inline uint32_t get_link_speed(const struct ixy_device* dev) {
	return dev->get_link_speed(dev);
}

static inline struct mac_address get_mac_addr(const struct ixy_device* dev) {
	return dev->get_mac_addr(dev);
}

static inline void set_mac_addr(struct ixy_device* dev, struct mac_address mac) {
	dev->set_mac_addr(dev, mac);
}

// calls ixy_tx_batch until all packets are queued with busy waiting
static void ixy_tx_batch_busy_wait(struct ixy_device* dev, uint16_t queue_id, struct pkt_buf* bufs[], uint32_t num_bufs) {
	uint32_t num_sent = 0;
	while ((num_sent += ixy_tx_batch(dev, queue_id, bufs + num_sent, num_bufs - num_sent)) != num_bufs) {
		// busy wait
	}
}

// getters/setters for PCIe memory mapped registers
// this code looks like it's in need of some memory barrier intrinsics, but that's apparently not needed on x86
// dpdk has release/acquire memory order calls before/after the memory accesses, but they are defined as
// simple compiler barriers (i.e., the same empty asm with dependency on memory as here) on x86
// dpdk also defines an additional relaxed load/store for the registers that only uses a volatile access,  we skip that for simplicity

static inline void set_reg32(uint8_t* addr, int reg, uint32_t value) {
	__asm__ volatile ("" : : : "memory");
	*((volatile uint32_t*) (addr + reg)) = value;
}

static inline uint32_t get_reg32(const uint8_t* addr, int reg) {
	__asm__ volatile ("" : : : "memory");
	return *((volatile uint32_t*) (addr + reg));
}

static inline void set_flags32(uint8_t* addr, int reg, uint32_t flags) {
	set_reg32(addr, reg, get_reg32(addr, reg) | flags);
}

static inline void clear_flags32(uint8_t* addr, int reg, uint32_t flags) {
	set_reg32(addr, reg, get_reg32(addr, reg) & ~flags);
}

static inline void wait_clear_reg32(const uint8_t* addr, int reg, uint32_t mask) {
	__asm__ volatile ("" : : : "memory");
	uint32_t cur = 0;
	while (cur = *((volatile uint32_t*) (addr + reg)), (cur & mask) != 0) {
		debug("waiting for flags 0x%08X in register 0x%05X to clear, current value 0x%08X", mask, reg, cur);
		usleep(10000);
		__asm__ volatile ("" : : : "memory");
	}
}

static inline void wait_set_reg32(const uint8_t* addr, int reg, uint32_t mask) {
	__asm__ volatile ("" : : : "memory");
	uint32_t cur = 0;
	while (cur = *((volatile uint32_t*) (addr + reg)), (cur & mask) != mask) {
		debug("waiting for flags 0x%08X in register 0x%05X, current value 0x%08X", mask, reg, cur);
		usleep(10000);
		__asm__ volatile ("" : : : "memory");
	}
}

// getters/setters for pci io port resources

static inline void write_io32(int fd, uint32_t value, size_t offset) {
	if (pwrite(fd, &value, sizeof(value), offset) != sizeof(value))
		error("pwrite io resource");
	__asm__ volatile("" : : : "memory");
}

static inline void write_io16(int fd, uint16_t value, size_t offset) {
	if (pwrite(fd, &value, sizeof(value), offset) != sizeof(value))
		error("pwrite io resource");
	__asm__ volatile("" : : : "memory");
}

static inline void write_io8(int fd, uint8_t value, size_t offset) {
	if (pwrite(fd, &value, sizeof(value), offset) != sizeof(value))
		error("pwrite io resource");
	__asm__ volatile("" : : : "memory");
}

static inline uint32_t read_io32(int fd, size_t offset) {
	__asm__ volatile("" : : : "memory");
	uint32_t temp;
	if (pread(fd, &temp, sizeof(temp), offset) != sizeof(temp))
		error("pread io resource");
	return temp;
}

static inline uint16_t read_io16(int fd, size_t offset) {
	__asm__ volatile("" : : : "memory");
	uint16_t temp;
	if (pread(fd, &temp, sizeof(temp), offset) != sizeof(temp))
		error("pread io resource");
	return temp;
}

static inline uint8_t read_io8(int fd, size_t offset) {
	__asm__ volatile("" : : : "memory");
	uint8_t temp;
	if (pread(fd, &temp, sizeof(temp), offset) != sizeof(temp))
		error("pread io resource");
	return temp;
}

#endif // IXY_DEVICE_H