src/pci.c

Walkthrough, interview notes & deep dive

Userspace PCI Orchestration

The pci.c file is the gateway for the ixy driver to escape the kernel's standard networking stack and take direct control of hardware. In a typical Linux environment, the kernel's ixgbe or virtio-net driver would claim the NIC. To implement a userspace driver, we must first "kidnap" the device from the kernel, configure its internal registers to allow userspace-initiated DMA, and finally map the hardware's register space into our process's memory. This file provides the low-level sysfs-based primitives required to achieve this handover.

Unbinding the Kernel Driver

The first step in taking control is ensuring the kernel is no longer managing the device. The function remove_driver interacts with the Linux PCI subsystem via the sysfs filesystem. By writing the PCI address (e.g., 0000:01:00.0) to the unbind file located at /sys/bus/pci/devices/.../driver/unbind, we force the kernel to release its grip on the device's resources. This prevents the kernel and our userspace driver from fighting over the same hardware registers, which would inevitably lead to a system crash or hardware undefined behavior.

void remove_driver(const char* pci_addr) {
	char path[PATH_MAX];
	snprintf(path, PATH_MAX, "/sys/bus/pci/devices/%s/driver/unbind", pci_addr);
	int fd = open(path, O_WRONLY);
	if (fd == -1) {
		debug("no driver loaded");
		return;
	}
	if (write(fd, pci_addr, strlen(pci_addr)) != (ssize_t) strlen(pci_addr)) {
		warn("failed to unload driver for device %s", pci_addr);
	}
	check_err(close(fd), "close");
}

Enabling Bus-Mastering

Once the device is free, we must explicitly enable its ability to perform DMA (Direct Memory Access). By default, if a device is not managed by a driver, its "Bus Master" capability may be disabled for security and stability reasons. The enable_dma function modifies the PCI Command Register. This register is located at offset 4 in the PCIe configuration space. We read the 16-bit register, set bit 2 (the Bus Master Enable bit as defined in the PCIe 3.0 specification), and write it back. Without this bit set, the NIC cannot initiate memory writes to our packet buffers, rendering it useless for high-speed reception.

void enable_dma(const char* pci_addr) {
	char path[PATH_MAX];
	snprintf(path, PATH_MAX, "/sys/bus/pci/devices/%s/config", pci_addr);
	int fd = check_err(open(path, O_RDWR), "open pci config");
	assert(lseek(fd, 4, SEEK_SET) == 4);
	uint16_t dma = 0;
	assert(read(fd, &dma, 2) == 2);
	dma |= 1 << 2;
	assert(lseek(fd, 4, SEEK_SET) == 4);
	assert(write(fd, &dma, 2) == 2);
	check_err(close(fd), "close");
}

Mapping the BAR0 Resource

The core of userspace driver performance is MMIO (Memory-Mapped I/O). The function pci_map_resource brings it all together. It first unbinds the kernel driver and enables DMA, then opens the resource0 file in sysfs. This file represents the device's Base Address Register 0 (BAR0), which usually contains the entire control register window for the NIC. By using mmap with MAP_SHARED, we map the hardware's physical registers directly into our virtual address space.

uint8_t* pci_map_resource(const char* pci_addr) {
	// ... unbind and enable_dma calls ...
	int fd = check_err(open(path, O_RDWR), "open pci resource");
	struct stat stat;
	check_err(fstat(fd, &stat), "stat pci resource");
	uint8_t* hw = (uint8_t*) check_err(mmap(NULL, stat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0), "mmap pci resource");
	check_err(close(fd), "close pci resource");
	return hw;
}

After this mapping is established, every pointer dereference we perform on the returned hw pointer translates directly to a PCIe TLP (Transaction Layer Packet) hitting the NIC hardware. This allows the driver to poll status registers or update tail pointers without the overhead of a system call. The companion function pci_open_resource provides a simpler wrapper for opening other device-related files in sysfs, such as MSX-X interrupt configurations or extended capability registers.

Interview angles

• What is Bus-Master DMA and why must we enable it? Bus-mastering allows a PCIe device to initiate data transfers on the bus independently of the CPU. For a NIC, this is critical because the hardware must be able to write incoming packets directly into RAM (RX) and read packets from RAM (TX) without the CPU manually copying every byte.

• Why is it mandatory to unbind the kernel driver before starting a userspace driver? If both the kernel and a userspace application attempt to manage the same hardware, they will conflict on register states and interrupt handling. Unbinding ensures the kernel's resource management (like the IOMMU or interrupt routing) is cleanly detached or prepared for handover.

• What is BAR0 and why is it mapped with MAP_SHARED? A Base Address Register (BAR) defines a window of physical addresses that the device responds to. BAR0 typically maps the NIC's internal registers. We use MAP_SHARED because MAP_PRIVATE would create a copy-on-write mapping of the sysfs file, whereas we need our writes to go directly to the hardware and our reads to reflect the hardware's current state.

Going deeper

uint16_t dma = 0;
assert(read(fd, &dma, 2) == 2);
dma |= 1 << 2;
assert(lseek(fd, 4, SEEK_SET) == 4);
assert(write(fd, &dma, 2) == 2);

The use of a read-modify-write (RMW) pattern here is mandatory because the PCI Command Register (offset 0x04) is a bitfield. Writing a hardcoded value would inadvertently toggle other critical controls, such as I/O Space (bit 0), Memory Space (bit 1), or Parity Error Response (bit 6). While PCI configuration space is strictly little-endian, this code performs a 2-byte read directly into a uint16_t. This works on x86 (little-endian) because the byte order in memory matches the bus order, but it would require le16toh on big-endian architectures like PowerPC to avoid bit-masking the wrong register flags.

struct stat stat;
check_err(fstat(fd, &stat), "stat pci resource");
uint8_t* hw = (uint8_t*) check_err(mmap(NULL, stat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0), "mmap pci resource");

Dynamically retrieving the mapping size via fstat on the sysfs resource file is safer than hardcoding BAR sizes. This st_size represents the actual aperture defined in the PCIe Base Address Register. The MAP_SHARED flag is critical; it ensures that stores to the hw pointer are visible to the hardware device rather than being trapped in a private process-local copy. However, the return type uint8_t* lacks volatile qualification. In a real driver, the compiler might optimize out back-to-back reads to the same status register or reorder a doorbell write unless the developer uses explicit memory barriers or volatile pointer casting.

Harder interview questions

• Q: Why does this code use `assert` for syscalls like `write` and `lseek`? A. It is a major reliability risk. assert is stripped when NDEBUG is defined for production builds. If that happens, the read, lseek, and write calls in enable_dma disappear entirely, and the driver will fail to enable bus-mastering without any error indication.
• Q: What happens if `resource0` is a prefetchable BAR? A. If the BAR is marked prefetchable, the CPU and PCIe bridges may perform write-combining or speculative reads. For NIC registers, we generally require Non-Prefetchable (Write-Through/Uncacheable) memory to ensure strict ordering and that every register poke hits the hardware immediately.
• Q: This code doesn't use VFIO or an IOMMU; what are the security implications? A. Without an IOMMU, the NIC performs DMA using raw physical addresses. A malicious or buggy userspace process could program the NIC to DMA into kernel memory or other processes' memory, as there is no hardware-level translation or protection (IOVA) between the device and system RAM.
• Q: Why does `remove_driver` ignore the error if `open` fails? A. If open fails, it likely means no kernel driver was bound to that PCI address in the first place (the driver symlink doesn't exist). In that case, the "unbind" step is unnecessary, so the function returns gracefully.

Gotchas

• String vs Binary: Writing to /sys/.../unbind requires the ASCII string of the PCI address (e.g., "0000:01:00.0"), whereas writing to /sys/.../config is a binary write to a raw offset.
• Resource Naming: This code assumes the NIC's primary register space is at resource0 (BAR0). If a device uses resource2 for MMIO and resource0 for I/O ports, this code will fail to map the registers correctly.
• Descriptor Leaks: pci_map_resource closes the file descriptor after mmap, which is fine, but the function provides no mechanism for munmap. Repeatedly initializing the device in a long-running process will eventually exhaust the process's virtual address space.

From ixy to a production driver

What the kernel path would do: ixy turns a BDF string into sysfs writes and an mmap() of resource0. A real in-kernel driver such as ixgbe.ko for Intel 82599-class NICs is matched by the PCI core through a struct pci_driver ID table and entered at .probe. In probe, the driver enables the function with pci_enable_device(), sets DMA ownership with pci_set_master(), claims BAR space with pci_request_regions() or pci_request_selected_regions(), and maps registers with pci_iomap()/ioremap-style helpers. That is the production version of ixy's enable_dma() plus pci_map_resource(): same PCI Command Register idea, but with kernel ownership, conflict checks, DMA masks, suspend state, and structured unwind, often using devm_ helpers.

The Bus Master bit is not the whole DMA story: ixy writes offset 0x04, bit 2, in PCI config space to enable Bus Mastering. That matters because pci_set_master() sets the same PCI_COMMAND_MASTER bit in the kernel. The missing half is DMA address control. Kernel drivers set DMA masks and use the DMA API so mappings are valid for the device and platform IOMMU. ixy's legacy path gives the NIC bus-master permission but creates no IOMMU domain, so a buggy or hostile device can DMA to arbitrary physical memory reachable by the platform.

How DPDK usually hands over devices: DPDK applications enter through rte_eal_init(), where EAL scans PCI devices, applies allow/block lists, and lets poll-mode drivers bind to matching devices. Administrators detach ports from ixgbe/i40e/other kernel drivers using dpdk-devbind.py or sysfs mechanisms such as driver_override, then bind them to vfio-pci. Modern DPDK strongly prefers vfio-pci because VFIO exposes device files only for IOMMU-isolated groups and maps BARs/DMA through that boundary. UIO paths like igb_uio or uio_pci_generic are closer to ixy's model: simple BAR mapping, much weaker isolation.

What ixy omits on purpose: It does no MSI-X allocation, AER recovery, hotplug handling, runtime PM or D-state transitions, SR-IOV VF/PF policy, or cleanup path to munmap() BAR0 and rebind the previous kernel driver. For a teaching driver, that keeps PCI handover visible in four small functions. For production, each omission is a reliability or containment issue: interrupts need vectors, error reporting needs reset logic, VFs need privilege boundaries, and exit paths must leave the host sane.

Virtio-net contrast: virtio-net over PCI is not just "BAR0 plus device registers." Modern virtio-pci discovers common, notify, ISR, device-specific, and PCI configuration structures through virtio PCI capabilities in config space; transitional devices also have legacy behavior. That makes BAR layout negotiation part of the device model, while ixy assumes one NIC-specific MMIO window and hands that directly to driver code.

Sources

• How To Write Linux PCI Drivers
• DPDK Linux Drivers
• dpdk-devbind Application
• Virtual I/O Device (VIRTIO) Version 1.2

Source

#include <assert.h>
#include <errno.h>
#include <linux/limits.h>
#include <stdio.h>
#include <sys/file.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>

#include "log.h"

void remove_driver(const char* pci_addr) {
	char path[PATH_MAX];
	snprintf(path, PATH_MAX, "/sys/bus/pci/devices/%s/driver/unbind", pci_addr);
	int fd = open(path, O_WRONLY);
	if (fd == -1) {
		debug("no driver loaded");
		return;
	}
	if (write(fd, pci_addr, strlen(pci_addr)) != (ssize_t) strlen(pci_addr)) {
		warn("failed to unload driver for device %s", pci_addr);
	}
	check_err(close(fd), "close");
}

void enable_dma(const char* pci_addr) {
	char path[PATH_MAX];
	snprintf(path, PATH_MAX, "/sys/bus/pci/devices/%s/config", pci_addr);
	int fd = check_err(open(path, O_RDWR), "open pci config");
	// write to the command register (offset 4) in the PCIe config space
	// bit 2 is "bus master enable", see PCIe 3.0 specification section 7.5.1.1
	assert(lseek(fd, 4, SEEK_SET) == 4);
	uint16_t dma = 0;
	assert(read(fd, &dma, 2) == 2);
	dma |= 1 << 2;
	assert(lseek(fd, 4, SEEK_SET) == 4);
	assert(write(fd, &dma, 2) == 2);
	check_err(close(fd), "close");
}

uint8_t* pci_map_resource(const char* pci_addr) {
	char path[PATH_MAX];
	snprintf(path, PATH_MAX, "/sys/bus/pci/devices/%s/resource0", pci_addr);
	debug("Mapping PCI resource at %s", path);
	remove_driver(pci_addr);
	enable_dma(pci_addr);
	int fd = check_err(open(path, O_RDWR), "open pci resource");
	struct stat stat;
	check_err(fstat(fd, &stat), "stat pci resource");
	uint8_t* hw = (uint8_t*) check_err(mmap(NULL, stat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0), "mmap pci resource");
	check_err(close(fd), "close pci resource");
	return hw;
}

int pci_open_resource(const char* pci_addr, const char* resource, int flags) {
	char path[PATH_MAX];
	snprintf(path, PATH_MAX, "/sys/bus/pci/devices/%s/%s", pci_addr, resource);
	debug("Opening PCI resource at %s", path);
	int fd = check_err(open(path, flags), "open pci resource");
	return fd;
}