src/app/ixy-pktgen.c

Walkthrough, interview notes & deep dive

ixy-pktgen.c serves as the primary performance benchmark for the ixy driver. It is a high-speed UDP packet generator designed to saturate a 10GbE link by minimizing the overhead of the memory management and descriptor ring updates. In the ixy datapath, this application bypasses the kernel entirely, interacting directly with the ixy_device to blast frames at wire speed.

Packet Templates and Checksums. The generator uses a static pkt_data array to define a standard UDP/IPv4 template. The PKT_SIZE is set to 60 bytes, which results in a 64-byte frame on the wire after the hardware appends the 4-byte CRC. To ensure the frames are valid for upstream switches, calc_ip_checksum implements a standard 16-bit one's complement sum over the 20-byte IP header. Because the packet headers (except for the sequence number) don't change, this checksum is calculated only once during initialization.

Memory Pre-filling Strategy. The init_mempool function demonstrates a clever optimization for high-speed transmission. Instead of formatting packets in the hot loop, ixy allocates a pool via memory_allocate_mempool and immediately pulls every buffer out using pkt_buf_alloc.

for (int buf_id = 0; buf_id < NUM_BUFS; buf_id++) {
	struct pkt_buf* buf = pkt_buf_alloc(mempool);
	buf->size = PKT_SIZE;
	memcpy(buf->data, pkt_data, sizeof(pkt_data));
	*(uint16_t*) (buf->data + 24) = calc_ip_checksum(buf->data + 14, 20);
	bufs[buf_id] = buf;
}

Once the buffers are initialized with the template and the IP checksum is written to offset 24, they are returned to the pool with pkt_buf_free. When the main loop later calls pkt_buf_alloc_batch, it receives buffers that are already "warm" and pre-formatted, requiring only a single 4-byte write for the sequence number.

The Transmission Loop. Inside main, the application initializes the hardware via ixy_init. The "hot loop" is built around BATCH_SIZE (64 packets). This batching is critical; it amortizes the cost of doorbell register writes across multiple packets. The code calls pkt_buf_alloc_batch to grab descriptors, updates the seq_num at the end of the payload, and then hands the batch to the driver.

while (true) {
	pkt_buf_alloc_batch(mempool, bufs, BATCH_SIZE);
	for (uint32_t i = 0; i < BATCH_SIZE; i++) {
		*(uint32_t*)(bufs[i]->data + PKT_SIZE - 4) = seq_num++;
	}
	ixy_tx_batch_busy_wait(dev, 0, bufs, BATCH_SIZE);
}

Async Transmission and Buffering. A key takeaway for low-level engineers is that ixy_tx_batch_busy_wait does not mean the packet is physically on the wire when the function returns. It means the descriptors have been posted to the NIC's hardware ring. The NIC will read these via DMA asynchronously. This is why we cannot immediately reuse a pkt_buf; we must wait for the hardware to set the "Done" (DD) bit in the descriptor, a process handled internally by the mempool and driver logic.

Throttled Telemetry. To avoid wasting CPU cycles on string formatting and I/O, the application uses a bitmask check (counter++ & 0xFFF) == 0 to throttle statistics. Every 4096 batches, it checks the monotonic_time. If a second has passed, it calls ixy_read_stats and print_stats_diff to output the current Mpps (Millions of packets per second) and Mbit/s throughput.

Interview angles

• Why use batching of 64 packets? Every time the driver notifies the NIC of new packets, it must write to a PCIe register (the Tail Pointer). These MMIO writes are expensive "uncacheable" operations that stall the CPU pipeline. Batching allows us to perform one MMIO write for 64 packets, significantly increasing the Mpps ceiling.

• How does ixy handle the checksum offload? In this specific pktgen implementation, the IP checksum is calculated in software during init_mempool. However, modern NICs like the 82599 (ixgbe) support hardware checksum offload. An engineer might be asked how to modify the descriptors to let the NIC calculate the checksum, which involves setting specific flags in the TX context descriptor.

• What is the significance of the 60-byte packet size? This is the minimum Ethernet frame size (excluding the 4-byte FCS/CRC). Small packets are the "worst-case" scenario for routers and NICs because the overhead per packet (preamble, inter-packet gap, descriptor processing) is highest relative to the data moved. If a driver can handle "64-byte wire size" at line rate, it can handle any size.

• Why does the code allocate new buffers instead of reusing the same array? Because TX is asynchronous. The NIC reads the buffer via DMA long after the ixy_tx function returns. If you modified the buffer immediately, you would cause a race condition where the NIC might transmit partially updated data or corrupted headers. The mempool ensures we only reuse buffers the hardware has explicitly finished with.

Going deeper

if (len % 1) error("odd-sized checksums NYI");
uint32_t cs = 0;
for (uint32_t i = 0; i < len / 2; i++) {
    cs += ((uint16_t*)data)[i];
    if (cs > 0xFFFF) {
        cs = (cs & 0xFFFF) + 1; // 16 bit one's complement
    }
}

The len % 1 check is a logic bug; any integer modulo 1 is 0, meaning the "odd-sized" guard is unreachable. Standard RFC 1071 checksums require padding a trailing odd byte to 16 bits. Additionally, the carry fold if (cs > 0xFFFF) is performed inside the loop. While technically correct for short 20-byte IP headers, it is less efficient than the standard practice of accumulating carries in a 32-bit or 64-bit integer and performing a final while (sum >> 16) fold outside the loop.

*(uint32_t*)(bufs[i]->data + PKT_SIZE - 4) = seq_num++;

This line injects a 4-byte sequence number into the end of the packet payload. It assumes the pointer data + 56 (since PKT_SIZE is 60) is 4-byte aligned. If the buffer is 64-byte aligned (typical for DMA), this offset is safe. However, casting uint8_t* to uint32_t* violates strict aliasing rules in C. On x86, the hardware handles the potential unaligned access, but on stricter architectures like ARM, this could trigger a SIGBUS or a costly trap to the kernel for alignment fixup.

Harder interview questions

• Q: Why does the code use a busy-wait transmit rather than checking for available space? ixy_tx_batch_busy_wait simplifies the logic by spinning until the NIC accepts the full batch. In high-performance systems, this avoids the complexity of a software-side "retry queue," but it can lead to head-of-line blocking if the link is saturated or the peer is exerting flow control (e.g., via Ethernet PAUSE frames).

• Q: What is the performance impact of the sequence number write? The write "dirties" the cache line just before the NIC reads it. In modern Intel systems with Data Direct I/O (DDIO), the NIC reads directly from the L3 cache. This write-before-send pattern is optimal because the data is likely still in the L1/L2 cache of the core, ensuring the DMA transfer doesn't need to hit main memory.

• Q: How does the memory pre-filling strategy interact with the NIC's async DMA? By pre-filling all 2048 buffers in init_mempool and then returning them to the pool, the application ensures that pkt_buf_alloc_batch always returns "warm" buffers with headers already set. This minimizes CPU cycles in the hot loop, but it relies on the fact that the NIC will never modify the buffer data during its read-only DMA transmit process.

Gotchas

• Endianness Mismatch: The manual length packing (>> 8 and & 0xFF) correctly sets Big-Endian for the IP header, but the seq_num injection via uint32_t* cast uses Host Endianness. On an x86 generator, the sequence number will appear Little-Endian in the payload, which may confuse capture tools expecting network byte order.
• Checksum Stale-ness: The IP checksum is calculated once during pre-fill. If you were to modify a flow-specific field like the Destination IP inside the while loop to simulate multiple flows, the packet would be dropped by any standard stack because the checksum at offset 24 is never updated.

From ixy to a production driver

ixy-pktgen.c is the smallest useful TX benchmark: one queue, one static 60-byte UDP/IPv4 template, 2048 warm buffers, 64 descriptors per burst, and a loop that changes only a payload sequence number before ixy_tx_batch_busy_wait(dev, 0, bufs, 64). Production generators keep the batching idea, but manage the parts ixy hides.

• Transmit completion and backpressure: ixy posts to queue 0 and busy-waits until the batch is accepted. An ixgbe-class driver tracks TDH/TDT, descriptor write-back, and the TX descriptor DD bit so completed packets can be unmapped and freed. Linux does this in ixgbe_clean_tx_irq, with queue stop/wake behavior when the ring is full. DPDK users of rte_eth_tx_burst handle short returns by retrying or freeing unsent mbufs. "TX is async" means ownership, cleanup, and bounded retry policy.

• Offloads instead of software checksums: this file writes the IPv4 header checksum once with calc_ip_checksum and leaves UDP checksum zero. Real drivers program per-packet metadata. On 82599, advanced TX descriptors and TX context descriptors request checksum insertion with TXSM and IXSM; Linux advertises NETIF_F_HW_CSUM, NETIF_F_IP_CSUM, NETIF_F_TSO, and related flags. DPDK mbufs carry IP/L4 checksum and TSO/LSO flags. Production code must translate packet intent into descriptor context, including MSS and header lengths.

• Scaling beyond one hot loop: ixy initializes one TX queue and one RX queue and always transmits on queue 0. Real pktgen, pktgen-DPDK, l2fwd, TRex, virtio-net multiqueue, and kernel drivers spread work across queues bound to cores. Receive steering uses RSS tables such as RETA; transmit selection uses DPDK per-core queues or kernel logic such as netdev_pick_tx and XPS. The concurrency question is single-owner lockless queues versus shared locked queues.

• Doorbells, interrupts, and congestion: ixy's batch size of 64 amortizes descriptor setup and TDT tail writes. Poll-mode ixy and DPDK avoid hot-path interrupts; kernel ixgbe uses NAPI, interrupt moderation via EITR/ITR, and TX completion interrupts or poll cleanup. Ethernet PAUSE/802.3x can stop a congested link and cause head-of-line blocking, so production generators report occupancy, drops, retries, and offered versus transmitted load.

• Operational robustness: ixy is intentionally tiny, around the thousand-line educational-driver scale described by the project, so it omits link-state changes, hotplug, power management, reset recovery, ethtool counters, hardware error stats, watchdog TX-hang detection, and most locking. That is the point: it exposes the minimum fast path. For an AMD/Solarflare interview, each omission prompts discussion of the OS, DMA memory, NIC rings, interrupts, offloads, and observability.

Sources

• Linux kernel: Scaling in the Networking Stack (RSS, RETA, XPS)
• DPDK ixgbe Poll Mode Driver (82599 offloads)
• Linux ixgbe driver source
• The Case for Writing Network Drivers in High-Level Programming Languages

Source

#include <stdio.h>

#include "stats.h"
#include "log.h"
#include "memory.h"
#include "driver/device.h"

// number of packets sent simultaneously to our driver
static const uint32_t BATCH_SIZE = 64;

// excluding CRC (offloaded by default)
#define PKT_SIZE 60

static const uint8_t pkt_data[] = {
	0x01, 0x02, 0x03, 0x04, 0x05, 0x06, // dst MAC
	0x10, 0x10, 0x10, 0x10, 0x10, 0x10, // src MAC
	0x08, 0x00,                         // ether type: IPv4
	0x45, 0x00,                         // Version, IHL, TOS
	(PKT_SIZE - 14) >> 8,               // ip len excluding ethernet, high byte
	(PKT_SIZE - 14) & 0xFF,             // ip len exlucding ethernet, low byte
	0x00, 0x00, 0x00, 0x00,             // id, flags, fragmentation
	0x40, 0x11, 0x00, 0x00,             // TTL (64), protocol (UDP), checksum
	0x0A, 0x00, 0x00, 0x01,             // src ip (10.0.0.1)
	0x0A, 0x00, 0x00, 0x02,             // dst ip (10.0.0.2)
	0x00, 0x2A, 0x05, 0x39,             // src and dst ports (42 -> 1337)
	(PKT_SIZE - 20 - 14) >> 8,          // udp len excluding ip & ethernet, high byte
	(PKT_SIZE - 20 - 14) & 0xFF,        // udp len exlucding ip & ethernet, low byte
	0x00, 0x00,                         // udp checksum, optional
	'i', 'x', 'y'                       // payload
	// rest of the payload is zero-filled because mempools guarantee empty bufs
};

// calculate a IP/TCP/UDP checksum
static uint16_t calc_ip_checksum(uint8_t* data, uint32_t len) {
	if (len % 1) error("odd-sized checksums NYI"); // we don't need that
	uint32_t cs = 0;
	for (uint32_t i = 0; i < len / 2; i++) {
		cs += ((uint16_t*)data)[i];
		if (cs > 0xFFFF) {
			cs = (cs & 0xFFFF) + 1; // 16 bit one's complement
		}
	}
	return ~((uint16_t) cs);
}

static struct mempool* init_mempool() {
	const int NUM_BUFS = 2048;
	struct mempool* mempool = memory_allocate_mempool(NUM_BUFS, 0);
	// pre-fill all our packet buffers with some templates that can be modified later
	// we have to do it like this because sending is async in the hardware; we cannot re-use a buffer immediately
	struct pkt_buf* bufs[NUM_BUFS];
	for (int buf_id = 0; buf_id < NUM_BUFS; buf_id++) {
		struct pkt_buf* buf = pkt_buf_alloc(mempool);
		buf->size = PKT_SIZE;
		memcpy(buf->data, pkt_data, sizeof(pkt_data));
		*(uint16_t*) (buf->data + 24) = calc_ip_checksum(buf->data + 14, 20);
		bufs[buf_id] = buf;
	}
	// return them all to the mempool, all future allocations will return bufs with the data set above
	for (int buf_id = 0; buf_id < NUM_BUFS; buf_id++) {
		pkt_buf_free(bufs[buf_id]);
	}

	return mempool;
}

int main(int argc, char* argv[]) {
	if (argc != 2) {
		printf("Usage: %s <pci bus id>\n", argv[0]);
		return 1;
	}

	struct ixy_device* dev = ixy_init(argv[1], 1, 1, 0);
	struct mempool* mempool = init_mempool();

	uint64_t last_stats_printed = monotonic_time();
	uint64_t counter = 0;
	struct device_stats stats_old, stats;
	stats_init(&stats, dev);
	stats_init(&stats_old, dev);
	uint32_t seq_num = 0;

	// array of bufs sent out in a batch
	struct pkt_buf* bufs[BATCH_SIZE];

	// tx loop
	while (true) {
		// we cannot immediately recycle packets, we need to allocate new packets every time
		// the old packets might still be used by the NIC: tx is async
		pkt_buf_alloc_batch(mempool, bufs, BATCH_SIZE);
		for (uint32_t i = 0; i < BATCH_SIZE; i++) {
			// packets can be modified here, make sure to update the checksum when changing the IP header
			*(uint32_t*)(bufs[i]->data + PKT_SIZE - 4) = seq_num++;
		}
		// the packets could be modified here to generate multiple flows
		ixy_tx_batch_busy_wait(dev, 0, bufs, BATCH_SIZE);

		// don't check time for every packet, this yields +10% performance :)
		if ((counter++ & 0xFFF) == 0) {
			uint64_t time = monotonic_time();
			if (time - last_stats_printed > 1000 * 1000 * 1000) {
				// every second
				ixy_read_stats(dev, &stats);
				print_stats_diff(&stats, &stats_old, time - last_stats_printed);
				stats_old = stats;
				last_stats_printed = time;
			}
		}
		// track stats
	}
}