src/driver/device.c

Walkthrough, interview notes & deep dive

The Role of device.c in ixy The driver/device.c file serves as the top-level hardware abstraction layer and entry point for the ixy userspace driver. In a low-level systems context, this file implements a factory pattern that allows the driver to support multiple hardware backends (Intel ixgbe and VirtIO) through a unified interface. By decoupling hardware identification from the core logic, ixy enables applications to remain device-agnostic while operating directly on hardware registers and DMA queues.

Probing the PCI Configuration Space The heart of this file is the ixy_init function. Before any packet processing can occur, the driver must identify the hardware sitting at a specific PCI address. It does this by accessing the PCI configuration space, a standardized set of registers provided by every PCI device for identification and configuration.

int config = pci_open_resource(pci_addr, "config", O_RDONLY);
uint16_t vendor_id = read_io16(config, 0);
uint16_t device_id = read_io16(config, 2);
uint32_t class_id = read_io32(config, 8) >> 24;
close(config);

The code uses pci_open_resource to open the config sysfs attribute for the device. It then performs surgical reads: read_io16 at offset 0 retrieves the 16-bit Vendor ID, and offset 2 retrieves the Device ID. A crucial sanity check is performed on the class_id (retrieved via read_io32 at offset 8 and shifted). In the PCI specification, a Class ID of 0x02 indicates a Network Controller. If the device is not a NIC, ixy_init calls error(), preventing the driver from attempting to talk to incompatible hardware.

Backend Dispatch and Initialization Once the IDs are fetched, the function performs a dispatch based on the Vendor ID. Specifically, it looks for 0x1af4, which is the registered Vendor ID for Red Hat/VirtIO devices. This allows the driver to distinguish between a physical Intel NIC and a virtualized NIC in a cloud environment.

if (vendor_id == 0x1af4 && device_id >= 0x1000) {
    return virtio_init(pci_addr, rx_queues, tx_queues);
} else {
    // Our best guess is to try ixgbe
    return ixgbe_init(pci_addr, rx_queues, tx_queues, interrupt_timeout);
}

If the device is a VirtIO NIC, it delegates to virtio_init. Otherwise, it assumes the device is an Intel 82599 family NIC and calls ixgbe_init. This fallback mechanism reflects the driver's primary focus on high-performance Intel hardware while maintaining extensibility.

Vtable Mechanism and MMIO The function returns a pointer to a struct ixy_device. In C, this is used to implement a vtable (virtual method table). The ixy_device struct contains function pointers for hardware-specific operations. When ixgbe_init or virtio_init runs, they allocate their own private state, map the hardware's BAR0 (Base Address Register) into the process's address space via MMIO, and then populate the ixy_device function pointers with their specific implementations.

This architecture allows the datapath to call dev->rx_batch(...) without knowing which driver is underneath. The performance cost is a single indirect function call, which is the standard trade-off for polymorphism in performance-critical C code.

Interview angles

• How does the OS expose PCI configuration space to userspace in Linux? Through sysfs (typically /sys/bus/pci/devices/.../config), which the driver opens and reads like a standard file to avoid needing a kernel-space helper for identification.
• Why check the class_id instead of just the vendor_id? A single vendor makes many types of devices (storage, graphics, network); checking the Class ID ensures the target device actually supports the networking operations the driver expects.
• What is the benefit of implementing a vtable in a userspace driver? it provides a clean Hardware Interface Layer (HIL), allowing the same application code to run on different NICs while still benefiting from zero-copy DMA and userspace MMIO performance.

Going deeper

uint32_t class_id = read_io32(config, 8) >> 24;
if (class_id != 2) {
    error("Device %s is not a NIC", pci_addr);
}

The PCI Class Code starts at offset 9, but ixy reads a 32-bit word from offset 8 using read_io32. Shifting right 24 bits isolates the Base Class byte. A value of 0x02 identifies a Network Controller. The read_io32 helper ensures host-byte order alignment, so the Base Class always occupies the MSB regardless of whether the CPU is little or big-endian. If the check fails, error() terminates the process immediately.

int config = pci_open_resource(pci_addr, "config", O_RDONLY);
...
close(config);
if (vendor_id == 0x1af4 && device_id >= 0x1000) {
    return virtio_init(pci_addr, rx_queues, tx_queues);
} else {
    return ixgbe_init(pci_addr, rx_queues, tx_queues, interrupt_timeout);
}

The pci_open_resource function opens the sysfs configuration space with O_RDONLY. This descriptor is closed via close(config) before dispatching to virtio_init or ixgbe_init, as configuration probing is independent of the MMIO BAR mapping used for data transfer. The 0x1af4 check identifies VirtIO devices, while the device_id >= 0x1000 guard captures both legacy and modern IDs.

Harder interview questions

• Why close the config file descriptor before driver initialization? - Configuration access via sysfs is distinct from the MMIO BAR mapping; closing it avoids resource leaks once identification is cached.
• What are the risks of the default ixgbe fallback? - It bypasses vendor verification (Intel is 0x8086), meaning a non-Intel NIC will likely cause a segmentation fault when the driver writes to invalid register offsets.
• How does error() impact the library's utility? - Because it calls exit(), it prevents the caller from implementing a "try-probing" loop or performing graceful fallbacks to other drivers.

Gotchas

• The silent fallback to ixgbe_init for any non-VirtIO hardware leads to cryptic crashes if the user provides a PCI address for an incompatible vendor.
• Using pci_open_resource on sysfs assumes world-readable permissions, which often fail in restricted environments or when VFIO group permissions are not correctly set.

From ixy to a production driver

In production, device.c's "look at PCI config space, choose a backend, return a vtable" role is split across the bus, driver core, and NIC driver. A Linux PCI NIC driver does not usually open /sys/bus/pci/devices/.../config from userspace and guess. It declares a struct pci_driver with an exact id_table, registers it with pci_register_driver(), and lets the PCI core call .probe(struct pci_dev *, const struct pci_device_id *) only for devices that match and are not already owned by another driver. The real ixgbe driver has an ixgbe_pci_tbl[] containing many Intel device IDs, exports it with MODULE_DEVICE_TABLE(pci, ixgbe_pci_tbl), and wires .probe = ixgbe_probe, .remove = ixgbe_remove, .shutdown, and PM callbacks in ixgbe_driver. That module table is what lets udev/module autoload bind ixgbe.ko to an 82599/X520-class adapter instead of relying on a silent "not virtio means ixgbe" fallback.

For virtio, the match is also table/spec driven. The virtio PCI spec uses vendor ID 0x1AF4; modern non-transitional virtio PCI devices use device IDs 0x1040 through 0x107f, while transitional legacy devices use 0x1000 through 0x103f. A real virtio-net stack distinguishes the network device type and virtio feature negotiation path instead of treating all 0x1af4 devices above 0x1000 as network cards. Interview point: the PCI class code 0x02 is only a broad "network controller" filter; production binding is by vendor/device/subsystem/class masks and then by device-specific initialization.

DPDK is closer to ixy's userspace shape, but still has a driver model. EAL scans buses, creates rte_pci_device objects, and registered PMDs expose an rte_pci_driver with a NULL-terminated rte_pci_id table plus probe/remove callbacks. Devices normally must be detached from kernel netdev drivers and bound to vfio-pci or a UIO driver using dpdk-devbind.py; VFIO also brings IOMMU isolation and file descriptors rather than assuming world-readable sysfs config and mmap access.

• ixy_init() has no hotplug add/remove story, no reference counting on a kernel pci_dev, and no equivalent of .remove to quiesce DMA before memory is freed.
• It has no driver coexistence policy beyond the user supplying one BDF; Linux and DPDK maintain global device/driver lists and avoid double ownership.
• It omits PM, reset, shutdown, SR-IOV, and PCI error-recovery hooks. In an interview, connect those hooks to real failure modes: surprise removal, FLR/reset after firmware wedging, suspend/resume, DMA faults, and graceful teardown of queues and interrupts.

Sources

• How To Write Linux PCI Drivers
• Linux ixgbe_main.c source
• DPDK Linux Drivers Guide
• Virtual I/O Device (VIRTIO) Version 1.0

Source

#include <sys/file.h>

#include "device.h"
#include "driver/ixgbe.h"
#include "driver/virtio.h"
#include "pci.h"

struct ixy_device* ixy_init(const char* pci_addr, uint16_t rx_queues, uint16_t tx_queues, int interrupt_timeout) {
	// Read PCI configuration space
	// For VFIO, we could access the config space another way
	// (VFIO_PCI_CONFIG_REGION_INDEX). This is not needed, though, because
	// every config file should be world-readable, and here we
	// only read the vendor and device id.
	int config = pci_open_resource(pci_addr, "config", O_RDONLY);
	uint16_t vendor_id = read_io16(config, 0);
	uint16_t device_id = read_io16(config, 2);
	uint32_t class_id = read_io32(config, 8) >> 24;
	close(config);
	if (class_id != 2) {
		error("Device %s is not a NIC", pci_addr);
	}
	if (vendor_id == 0x1af4 && device_id >= 0x1000) {
		return virtio_init(pci_addr, rx_queues, tx_queues);
	} else {
		// Our best guess is to try ixgbe
		return ixgbe_init(pci_addr, rx_queues, tx_queues, interrupt_timeout);
	}
}