src/app/ixy-fwd.c

The forwarding example — the whole datapath is this poll loop.

Walkthrough, interview notes & deep dive

The Userspace Datapath as a Poll Loop

The ixy-fwd.c file implements a high-performance packet forwarding engine using the ixy userspace NIC driver. It sits at the very top of the datapath, orchestrating the movement of packets between two network interfaces. Unlike traditional kernel drivers that rely on interrupts, this application uses a pure busy-poll model. This eliminates the overhead of context switches and interrupt handling latency, making it ideal for low-latency requirements found in Solarflare or AMD networking environments.

Initialization and Configuration

The execution starts in main(), where the environment is set up. The driver uses ixy_init() to map the hardware registers into the process address space via MMIO and set up DMA memory pools.

struct ixy_device* dev1 = ixy_init(argv[1], 1, 1, -1);
struct ixy_device* dev2 = ixy_init(argv[2], 1, 1, 0);

These calls initialize the PCI devices, allocate descriptor rings, and prepare the NIC for operation. The parameters specify the PCI address, the number of RX/TX queues, and the NUMA node for memory allocation. Ensuring memory is local to the CPU socket is critical for minimizing QPI/UPI interconnect traffic and maintaining high throughput.

The forwarding poll loop: rx_batch then forward then tx_batch

The Forwarding Core

The heart of the application is the forward() function. It operates on an array of packet buffers, bufs[], with a size defined by BATCH_SIZE (set to 32). This batching is a fundamental optimization: it amortizes the cost of hardware register writes and descriptor ring management across multiple packets.

In the RX phase, ixy_rx_batch() scans the hardware's RX descriptor ring for "done" packets. The driver returns pointers to pkt_buf structures already populated by the NIC via DMA.

RX hot path: scan the descriptor ring, refill buffers

Once received, the code performs a "touch" operation: bufs[i]->data[1]++. This forces the CPU to bring the packet header into the L1/L2 cache. This is necessary for realistic benchmarking; otherwise, the packet might simply sit in the L3 cache or main memory, hiding the true cost of memory access in a real application.

Transmission and Memory Management

After processing, packets are handed to ixy_tx_batch(). This function places pointers to the buffers onto the TX descriptor ring and updates the hardware "Tail" register via an MMIO write to notify the NIC that new packets are ready for transmission.

TX hot path: clean done descriptors, post new ones

A crucial detail in forward() is how it handles transmission failures. If ixy_tx_batch() cannot send all received packets (e.g., if the TX ring is full), the remaining packets are freed using pkt_buf_free().

uint32_t num_tx = ixy_tx_batch(tx_dev, tx_queue, bufs, num_rx);
for (uint32_t i = num_tx; i < num_rx; i++) {
    pkt_buf_free(bufs[i]);
}

This "drop if full" strategy prevents the application from blocking and accumulating latency. In low-latency networking, it is often better to drop a packet than to delay the entire pipeline.

Performance Throttling for Statistics

The main loop runs as fast as possible, but printing statistics is an expensive operation. To avoid calling the system clock too frequently, the code uses a bitwise check: if ((counter++ & 0xFFF) == 0). This ensures that the time is only checked once every 4096 iterations. Only when this check passes and at least one second has elapsed does the driver read the hardware counters via ixy_read_stats() and print the throughput.

Interview angles

Why use a BATCH_SIZE of 32 instead of processing packets one by one?

Batching amortizes the fixed costs of MMIO writes (doorbells) and descriptor ring management. Each MMIO write is a costly operation that can stall the CPU pipeline; by updating the hardware tail pointer once for 32 packets, we significantly reduce the per-packet overhead.

What are the consequences of the busy-poll model used in main()?

The CPU core will show 100% utilization even if no traffic is flowing. This eliminates interrupt latency and "jitter" caused by context switching, but it is less power-efficient and requires dedicated cores for the datapath.

Why is pkt_buf_free() necessary for packets that fail to transmit?

In a userspace driver, memory management is manual. Every packet received via ixy_rx_batch() is "owned" by the software. If the software cannot pass that ownership back to the NIC via ixy_tx_batch(), it must explicitly free the buffer to the mempool, or the driver will eventually run out of descriptors/buffers and stop receiving traffic.

How does the NIC know where to put the packet data in memory?

Through DMA (Direct Memory Access). During ixy_init(), the driver allocates physically contiguous memory and writes the physical addresses into the descriptor rings. The NIC reads these descriptors and writes incoming packet data directly into those addresses without CPU intervention.

Going deeper

Packet touching and cache locality.

for (uint32_t i = 0; i < num_rx; i++) {
    bufs[i]->data[1]++;
}
uint32_t num_tx = ixy_tx_batch(tx_dev, tx_queue, bufs, num_rx);

The payload increment bufs[i]->data[1]++ is a deliberate "touch" to force the CPU to pull the packet header into the L1 cache. In a pure zero-copy forwarder, packet data might otherwise stay in the LLC or DRAM. This touch ensures the benchmark accounts for the "memory wall" penalty of actual software processing. Ownership of the bufs pointers transitions from the NIC RX descriptors to the application array upon a successful ixy_rx_batch.

Handling TX backpressure and memory safety.

for (uint32_t i = num_tx; i < num_rx; i++) {
    pkt_buf_free(bufs[i]);
}

If the TX descriptor ring is full, ixy_tx_batch returns num_tx < num_rx. This loop handles the leftovers. Since the NIC did not take ownership of these buffers, the application must explicitly call pkt_buf_free. Failing to do so causes a memory leak that eventually starves the RX side of available buffers in the mempool, leading to a total collapse of packet reception.

Optimized stats throttling via bitmask.

if ((counter++ & 0xFFF) == 0) {
    uint64_t time = monotonic_time();
    if (time - last_stats_printed > 1000 * 1000 * 1000) { ... }
}

Polling monotonic_time() (which involves a syscall or specialized instruction) every iteration is expensive. The bitwise mask & 0xFFF (once every 4096 iterations) is a high-performance alternative to the modulo operator. It ensures the datapath remains tight while still providing periodic stats updates roughly every second.

Harder interview questions

Q: How does the code handle single-port aliasing? A: The if (dev1 != dev2) check prevents redundant MMIO reads for statistics if the user provides the same PCI ID twice. Without this, ixy_read_stats would be called on the same registers twice, wasting cycles and potentially interfering with "clear-on-read" register logic.
Q: What happens if `dev2`'s link goes down? A: The TX ring will fill up immediately. The forward function will continue to "receive" packets from dev1 but will immediately drop them in the num_tx loop. This prevents the application from hanging but results in 100% loss for that direction.
Q: Why is the NUMA node specified as `-1` for `dev1` and `0` for `dev2`? A: Passing -1 tells the driver to auto-detect the optimal node (closest to the PCIe device), whereas 0 forces node 0. If dev2 is physically attached to socket 1, forcing node 0 will cause cross-socket QPI/UPI traffic, significantly increasing latency and reducing peak throughput.

Gotchas

Single-threaded blocking: Because both forward calls are in the same loop, any delay in processing one direction (e.g., a large RX batch) directly increases the latency for the other. The two directions are interleaved, not concurrent.
Idle stats delay: If the link is idle and the loop is throttled by other system processes, the 0xFFF gate might take significantly longer than 1 second to trigger, causing the statistics display to appear "frozen."
Payload corruption: The data[1]++ touch mutates the packet. Since the first 6 bytes of an Ethernet frame are the Destination MAC, this forwarder is technically corrupting addresses, making it unsuitable for actual network bridging.

From ixy to a production driver

Production polling is conditional: ixy-fwd.c is deliberately brutal: one thread spins forever, calls ixy_rx_batch(), modifies one byte, calls ixy_tx_batch(), and frees packets the TX ring cannot accept. In the in-kernel ixgbe path, receive work is normally interrupt-driven until load arrives. The interrupt handler schedules NAPI, disables or masks further RX interrupts for that queue, and ixgbe_poll drains a bounded amount of work before napi_complete_done() lets interrupts be re-enabled. That matters in interviews because NAPI is not "no polling"; it is adaptive interrupt mitigation. ixy burns a full core at 100% because its teaching goal is to expose the hot datapath and avoid scheduler, IRQ, and sk_buff complexity.

DPDK is the closer production cousin: DPDK l2fwd looks structurally similar: a run-to-completion loop calls rte_eth_rx_burst(), edits/forwards packets, and transmits with rte_eth_tx_buffer() plus periodic rte_eth_tx_buffer_flush(). The difference is not the existence of polling, but the amount of production machinery around it. DPDK maps ports and queues to lcores, pins those lcores, uses mempools and rte_mbuf, and buffers TX so occasional partial bursts do not immediately become drops. ixy's single thread handles both directions and both ports, which is easier to reason about but is not how a high-throughput forwarder would partition work.

Scaling starts at the queues: ixy_init(..., 1, 1) asks for one RX and one TX queue per NIC. A production 82599 deployment would use multiqueue, RSS hashing, the RSS redirection table (RETA) to steer flows to RX queues, and per-queue MSI-X vectors so different cores can poll independent queues. The ring mechanics are still recognizable: software consumes descriptors and updates NIC tail registers such as RDT on RX refill and TDT on TX submit. The interview point is that batching alone improves amortization, but queue parallelism is what lets the design scale across cores without locking the same ring.

Offloads and policy are intentionally absent: A real forwarder also decides when to enable checksum offload, TCP segmentation offload, receive coalescing features such as LRO/RSC, VLAN filtering, flow control, link-state handling, and richer stats through ethtool or PMD xstats. It also needs a policy for TX-full: retry, buffer, backpressure, drop by QoS class, or signal congestion. ixy just calls pkt_buf_free() for unsent buffers. That is acceptable here because the file is a microscope for descriptor rings and cache-friendly bursts, not a product dataplane. In an interview, call out that this simplicity is the point, then explain the missing robustness: link changes, queue ownership rules, synchronization if queues become shared, hardware errors, reset paths, and observability.

Sources

Source

filesrc/app/ixy-fwd.c

#include <stdio.h>

#include "stats.h"
#include "memory.h"
#include "driver/device.h"

const int BATCH_SIZE = 32;

static void forward(struct ixy_device* rx_dev, uint16_t rx_queue, struct ixy_device* tx_dev, uint16_t tx_queue) {
	struct pkt_buf* bufs[BATCH_SIZE];
	uint32_t num_rx = ixy_rx_batch(rx_dev, rx_queue, bufs, BATCH_SIZE);
	if (num_rx > 0) {
		// touch all packets, otherwise it's a completely unrealistic workload if the packet just stays in L3
		for (uint32_t i = 0; i < num_rx; i++) {
			bufs[i]->data[1]++;
		}
		uint32_t num_tx = ixy_tx_batch(tx_dev, tx_queue, bufs, num_rx);
		// there are two ways to handle the case that packets are not being sent out:
		// either wait on tx or drop them; in this case it's better to drop them, otherwise we accumulate latency
		for (uint32_t i = num_tx; i < num_rx; i++) {
			pkt_buf_free(bufs[i]);
		}
	}
}

int main(int argc, char* argv[]) {
	if (argc != 3) {
		printf("%s forwards packets between two ports.\n", argv[0]);
		printf("Usage: %s <pci bus id2> <pci bus id1>\n", argv[0]);
		return 1;
	}

	struct ixy_device* dev1 = ixy_init(argv[1], 1, 1, -1);
	struct ixy_device* dev2 = ixy_init(argv[2], 1, 1, 0);

	uint64_t last_stats_printed = monotonic_time();
	struct device_stats stats1, stats1_old;
	struct device_stats stats2, stats2_old;
	stats_init(&stats1, dev1);
	stats_init(&stats1_old, dev1);
	stats_init(&stats2, dev2);
	stats_init(&stats2_old, dev2);

	uint64_t counter = 0;
	while (true) {
		forward(dev1, 0, dev2, 0);
		forward(dev2, 0, dev1, 0);

		// don't poll the time unnecessarily
		if ((counter++ & 0xFFF) == 0) {
			uint64_t time = monotonic_time();
			if (time - last_stats_printed > 1000 * 1000 * 1000) {
				// every second
				ixy_read_stats(dev1, &stats1);
				print_stats_diff(&stats1, &stats1_old, time - last_stats_printed);
				stats1_old = stats1;
				if (dev1 != dev2) {
					ixy_read_stats(dev2, &stats2);
					print_stats_diff(&stats2, &stats2_old, time - last_stats_printed);
					stats2_old = stats2;
				}
				last_stats_printed = time;
			}
		}
	}
}