src/pci.h

PCI helper declarations.

Walkthrough, interview notes & deep dive

The ixy_pci.h header defines the low-level interface required to bootstrap a userspace network driver. It provides functions to interact with the Linux sysfs filesystem, allowing the driver to discover hardware, take control of the device from the kernel, and map the NIC's registers into the process memory space. The file uses a standard #ifndef include guard and includes stdint.h for fixed-width integer types.

#ifndef IXY_PCI_H
#define IXY_PCI_H

#include <stdint.h>

struct ixy_device;

Driver Unbinding and DMA. The remove_driver function is the first step in the initialization sequence. It unbinds any existing kernel driver (such as ixgbe) from the specified PCI address by writing to the unbind path in sysfs. Once the kernel releases the device, enable_dma is used to set the Bus Master bit in the PCI configuration space. This bit is a prerequisite for DMA operations, enabling the NIC to initiate memory transactions on the PCIe bus.

Resource Mapping. To control the hardware, the driver needs access to its MMIO (Memory Mapped I/O) registers. The functions pci_open_resource and pci_map_resource work together to locate and map these regions. Typically, the driver maps resource0 (corresponding to BAR0) to gain access to the device's control registers. This allows the userspace process to trigger doorbells or read status bits by simply dereferencing a pointer.

void remove_driver(const char* pci_addr);
void enable_dma(const char* pci_addr);
uint8_t* pci_map_resource(const char* bus_id);
int pci_open_resource(const char* pci_addr, const char* resource, int flags);

Interview angles

Why is bus-mastering critical for NIC performance? Bus-mastering allows the NIC to perform DMA (Direct Memory Access). This allows the hardware to transfer packet data directly to or from host memory without involving the CPU for every byte, which is essential for high-throughput, low-latency networking.
What is the purpose of unbinding the kernel driver? It prevents the Linux kernel from competing with the userspace driver for hardware access. If two drivers attempted to manage the same device's registers and interrupts simultaneously, it would lead to undefined behavior, crashes, or data corruption.
What role does BAR0 play in a userspace driver? Base Address Register 0 defines the memory region where the NIC's control registers are located. By mapping this region into its own address space using mmap, the userspace driver can perform I/O by simply reading or writing to specific memory addresses, bypassing slow system calls.

Going deeper

The implementation of enable_dma performs a precise read-modify-write on the PCI configuration space to avoid clobbering critical device state:

uint16_t status;
pread(fd, &status, 2, 4);
status |= (1 << 2);
pwrite(fd, &status, 2, 4);

Offset 4 targets the Command Register. We read 2 bytes (uint16_t) to preserve adjacent bits like I/O Space and Memory Space Enable. Setting bit 2 (Bus Master) is the hardware "go" signal; without it, the NIC cannot initiate DMA transactions to read descriptors or write packet data to host memory.

Resource mapping utilizes fstat to dynamically determine BAR size, ensuring compatibility across different NIC models without hardcoding offsets:

fstat(fd, &st);
uint8_t* mem = mmap(NULL, st.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
close(fd);

The MAP_SHARED flag is mandatory for MMIO; MAP_PRIVATE would trigger Copy-on-Write, causing register writes to vanish into a local RAM page instead of reaching the device. Note that the file descriptor is closed immediately after mmap because the kernel maintains the mapping's lifecycle until munmap is called.

Harder interview questions

Why is the returned uint8_t* dangerous without a volatile cast? The compiler may elide "redundant" register reads in a polling loop or reorder doorbell writes. You must cast the pointer or use compiler barriers to ensure every access reaches the PCIe bus in the correct order.
What happens if you forget to set the Bus Master bit? The CPU can still write to the NIC's registers (MMIO), but the NIC cannot "pull" descriptors or "push" packets. The driver will appear to function but will never actually move data, leading to a silent hang.
Why use string-based sysfs writes for unbinding instead of an ioctl? Sysfs provides a stable, filesystem-level ABI that doesn't require complex version-specific headers, making it more portable for userspace drivers across different kernel versions.

Gotchas

The read-modify-write on the command register is not atomic. In a multi-process environment, concurrent configuration changes can result in lost bits or corrupted device states.
The remove_driver unbind operation is silent. If the PCI address is incorrect or the driver is already unbound, the write to unbind succeeds without error, potentially masking a failure to gain exclusive device access.

From ixy to a production driver

What this header really owns: vendor/ixy/src/pci.h is the tiny PCI bring-up boundary for ixy: detach the normal kernel owner with remove_driver(), flip DMA on with enable_dma(), then open and mmap() a BAR via pci_open_resource() / pci_map_resource(). For an Intel 82599-style NIC, resource0 is BAR0, the MMIO register window. The important register detail is concrete: the PCI Command register is at config-space offset 0x04, and Bus Master Enable is bit 2; without it the NIC cannot issue DMA reads/writes.

What Linux would do instead: an in-kernel driver such as ixgbe.ko or virtio-net is not expected to scrape sysfs. The PCI core matches IDs and calls the driver's probe method. The driver then follows the PCI API path: pci_enable_device() enables the function, pci_set_master() sets Bus Master Enable, pci_request_regions() claims BAR ownership, and pci_iomap() maps device registers. From there it sets up queues, MSI-X vectors, NAPI, and teardown paths. For DMA, it does not hand physical addresses to hardware casually: it uses the DMA API, for example dma_map_single(), so the kernel can program or respect the IOMMU and cache-coherency rules. Production drivers also carry the boring but interview-critical machinery: AER/error recovery, hotplug removal, reset flows including FLR where available, runtime power management and PCI D-states, locking, and ordered teardown.

What DPDK would do instead: a DPDK poll-mode driver is still userspace and polling-oriented like ixy, but device ownership normally goes through the DPDK PCI bus and a kernel helper driver. Modern DPDK recommends binding NICs to vfio-pci: VFIO owns the device, checks IOMMU groups, exposes BARs safely to userspace, and maps DPDK hugepage memory for DMA through an IOMMU-backed interface. Older UIO paths such as igb_uio or uio_pci_generic are simpler, but provide much weaker isolation. VFIO is the production-safe answer because a bad DMA address is constrained by the IOMMU mapping instead of becoming arbitrary host memory corruption.

What ixy deliberately leaves out:

remove_driver() writes sysfs bind/unbind files directly. That is easy to read and good for education, but it bypasses the lifetime, hotplug, and policy checks a real driver stack relies on.
enable_dma() performs a userspace read-modify-write of config offset 0x04 to set bit 2. There is no kernel-side serialization around that non-atomic update, no reset coordination, and no full PCI enable sequence.
pci_map_resource() maps sysfs resource0 directly. That is enough to touch BAR0 MMIO registers, but unlike VFIO it does not create DMA isolation.
ixy has no MSI/MSI-X interrupt path because it is a pure poll-mode driver, no AER recovery, no power-management callbacks, no FLR choreography, and no hotplug story.

Why interviewers care: this file is a clean way to test whether you know the difference between "I can make the NIC move packets" and "I can safely own a PCIe device in a hostile, concurrent OS." ixy removes layers to make the hardware visible: BAR0 is just a mapped register file, Bus Master Enable is just a bit, and unbinding is just sysfs. A production answer must immediately add back ownership, isolation, DMA mapping, interrupts or deliberate polling, reset/error recovery, and power/hotplug behavior.

Sources

Source

filesrc/pci.h

#ifndef IXY_PCI_H
#define IXY_PCI_H

#include <stdint.h>

struct ixy_device;

void remove_driver(const char* pci_addr);
void enable_dma(const char* pci_addr);
uint8_t* pci_map_resource(const char* bus_id);
int pci_open_resource(const char* pci_addr, const char* resource, int flags);

#endif // IXY_PCI_H