src/app/ixy-pcap.c

Walkthrough, interview notes & deep dive

The Role of ixy-pcap ixy-pcap.c is a small utility app in the ixy project that captures live traffic from a userspace driver and writes it to disk in the standard .pcap format. It sits at the end of the RX datapath, consuming buffers straight from the NIC receive rings and bypassing the kernel network stack. It is primarily pedagogical: a clear example of how raw hardware buffers become standard trace files.

Logic Walkthrough The app follows the usual poll-mode driver lifecycle. It calls ixy_init to map the NIC PCI resources into userspace and bring up the hardware, requesting one RX queue and one TX queue (only RX is used here).

struct ixy_device* dev = ixy_init(argv[1], 1, 1, 0);
FILE* pcap = fopen(argv[2], "wb");
pcap_hdr_t header = {
    .magic_number = 0xa1b2c3d4,
    .version_major = 2,
    .version_minor = 4,
    .snaplen = 65535,
    .network = 1, // Ethernet
};
fwrite(&header, sizeof(header), 1, pcap);

The core execution resides in a while loop that continues until the requested n_packets count is met. Inside this loop, ixy_rx_batch polls the hardware for a batch of packets, constrained by BATCH_SIZE (32). For every packet received, the code captures a timestamp via gettimeofday, populates a pcaprec_hdr_t, and performs two fwrite calls: one for the record metadata and one for the raw frame data stored in bufs[i]->data. Crucially, after the data is written, pkt_buf_free is called to return the buffer to the driver mempool, preventing descriptor exhaustion.

The PCAP Mechanism The application implements the libpcap file format manually to avoid external dependencies. The process begins with a global header defined by pcap_hdr_t. This includes the magic number 0xa1b2c3d4, which indicates a standard pcap file with microsecond resolution. The network field is set to 1, signifying that the link layer is Ethernet.

while (n_packets != 0) {
    uint32_t num_rx = ixy_rx_batch(dev, 0, bufs, BATCH_SIZE);
    struct timeval tv;
    gettimeofday(&tv, NULL);
    for (uint32_t i = 0; i < num_rx && n_packets != 0; i++) {
        pcaprec_hdr_t rec_header = {
            .ts_sec = tv.tv_sec,
            .ts_usec = tv.tv_usec,
            .incl_len = bufs[i]->size,
            .orig_len = bufs[i]->size
        };
        fwrite(&rec_header, sizeof(pcaprec_hdr_t), 1, pcap);
        fwrite(bufs[i]->data, bufs[i]->size, 1, pcap);
        pkt_buf_free(bufs[i]);
    }
}

Each packet is preceded by a pcaprec_hdr_t. Both incl_len (included length) and orig_len (original length) are set to bufs[i]->size, since the driver captures the full frame without truncation. Because ixy is poll-mode, the CPU spins on ixy_rx_batch for lowest latency, but the synchronous fwrite calls are a real bottleneck: a production logger would offload disk I/O to a separate thread or async ring.

Interview angles

• Q: What is the primary bottleneck in this specific implementation?
• A: The synchronous fwrite calls. In a userspace driver, the goal is to keep up with line rate (e.g., 10Gbps). Blocking on disk I/O inside the polling loop will cause the RX descriptors to fill up quickly, leading to hardware-level packet drops. A production-grade logger would use io_uring or a multi-threaded producer-consumer ring buffer to decouple RX from disk writes.

• Q: How does the app ensure buffer safety?
• A: A strict ownership model. ixy_rx_batch hands ownership of each pkt_buf to the app; the app must return it to the driver's mempool via pkt_buf_free. Skipping that leaks buffers and eventually starves RX as the hardware runs out of free descriptors.

• Q: Why `gettimeofday` instead of hardware timestamps?
• A: Software timestamping is used here for simplicity. For accurate latency analysis you would prefer NIC hardware timestamps (if exposed in the RX descriptor) to avoid the jitter between the packet hitting the wire and the CPU calling gettimeofday.

Going deeper

Mempool Leak on Termination The implementation contains a critical memory leak when a packet limit is specified.

for (uint32_t i = 0; i < num_rx && n_packets != 0; i++) {
    // ... write packet ...
    pkt_buf_free(bufs[i]);
    if (n_packets > 0) n_packets--;
}

If ixy_rx_batch retrieves 32 packets but n_packets hits zero after the 5th, the remaining 27 pkt_buf pointers in the bufs array are abandoned. Because ixy uses a fixed-size mempool, these leaked buffers are never returned to the hardware RX ring, eventually starving the NIC and halting all reception.

Host Endianness Punning The code writes raw structs directly to disk to define the PCAP format.

pcap_hdr_t header = { .magic_number = 0xa1b2c3d4, ... };
fwrite(&header, sizeof(header), 1, pcap);

The 0xa1b2c3d4 magic number tells readers the file is in the host's native endianness. While this allows for efficient fwrite calls without byte-swapping, it makes the binary file platform-dependent and assumes the compiler will not insert padding between the uint16_t and int32_t fields.

Harder interview questions

• Why batch 32 packets if the user might only want 1? Batching amortizes the overhead of the PCIe "doorbell" register write and memory-mapped I/O (MMIO) checks. Even if it complicates the cleanup logic, processing one packet at a time would be orders of magnitude slower due to CPU-to-NIC synchronization overhead.
• The loop limits writes by n_packets but ixy_rx_batch always pulls a full BATCH_SIZE. How would you bound capture precisely without leaking? Pass the remaining count down as the batch size (min(BATCH_SIZE, n_packets)), or free the untouched tail of bufs after the inner loop. Otherwise you over-fetch and abandon descriptors.

Gotchas

• Batch Timestamping: gettimeofday is called once per batch. If 32 packets arrive at 100Gbps, they are captured over microseconds but will all receive the exact same timestamp in the PCAP, masking inter-packet arrival jitter.
• NULL Dereference: If fopen fails, the error() macro is called. If the underlying library implementation of error does not call exit(), the program will attempt to fwrite to a NULL pointer, causing a segmentation fault.

From ixy to a production driver

ixy-pcap.c is the smallest useful version of a capture appliance: map the NIC with ixy_init, poll ixy_rx_batch, timestamp a batch with gettimeofday, write classic pcap records, then pkt_buf_free each buffer. Production capture keeps the same conceptual pipeline, but moves several responsibilities back into battle-tested kernel or framework paths.

With tcpdump/libpcap on Linux, capture normally enters through AF_PACKET/PF_PACKET packet sockets. High-rate capture uses PACKET_RX_RING with mmap; modern libpcap can select TPACKET_V3, where the kernel fills block-based rings and userspace drains completed blocks. That is still a kernel-driver path: the ixgbe driver receives from hardware, builds/skbs or packet-ring entries, applies in-kernel classic/eBPF filters when requested, and exposes packet data through the packet socket. ixy deliberately skips that machinery and reads NIC RX rings directly in userspace. The interview point is ownership and cost: ixy teaches descriptors, DMA buffers, and mempools; tcpdump teaches the kernel capture ABI, filtering, privileges, and portability.

Timestamping is the biggest semantic shortcut. ixy calls gettimeofday once after a batch arrives, so up to 32 packets can get the same microsecond timestamp and all include user-space scheduling and polling delay. Linux capture can request SO_TIMESTAMPING with software RX stamps, and, when the NIC and driver support it, hardware RX stamps configured through SIOCSHWTSTAMP for PTP-style accuracy. For Intel 82599/ixgbe, timestamp-related identifiers include IXGBE_RXDADV_STAT_TS in the advanced RX descriptor status and MMIO registers such as IXGBE_TSYNCRXCTL, IXGBE_RXSTMPL, and IXGBE_RXSTMPH; real code must respect validity bits and hardware filtering limits instead of assuming every packet has a stamp.

Classic pcap also has capture policy. snaplen is the maximum saved bytes per packet; real tools can truncate, setting captured length below original length. ixy sets both incl_len and orig_len to the full frame size, which is simple and good for teaching, but wastes disk and PCI/cache bandwidth when only headers are needed.

The synchronous fwrite calls are another intentional simplification. At 10G or higher, disk I/O in the poll loop will back up RX rings and drop packets. Production designs decouple capture from writing with rings, separate writer threads, or framework support. DPDK’s path is a useful comparison: rte_pdump mirrors packets from a primary process, while dpdk-dumpcap writes Pcapng using librte_pcapng.

Finally, ixy writes C structs for classic pcap directly. The magic 0xa1b2c3d4 marks microsecond-resolution classic pcap and lets readers infer byte order, but raw struct writes assume the local ABI layout and padding. Pcapng adds section/interface metadata, timestamp resolution, drop counters, and multi-interface support, which is why production capture files increasingly prefer it.

Sources

• Linux PACKET_MMAP documentation
• Linux networking timestamping documentation
• PCAP Capture File Format
• DPDK packet capture libraries and tools

Source

#include <stdio.h>
#include <unistd.h>
#include <sys/time.h>

#include "driver/device.h"

const int BATCH_SIZE = 32;

// From https://wiki.wireshark.org/Development/LibpcapFileFormat
typedef struct pcap_hdr_s {
	uint32_t magic_number;  /* magic number */
	uint16_t version_major; /* major version number */
	uint16_t version_minor; /* minor version number */
	int32_t  thiszone;      /* GMT to local correction */
	uint32_t sigfigs;       /* accuracy of timestamps */
	uint32_t snaplen;       /* max length of captured packets, in octets */
	uint32_t network;       /* data link type */
} pcap_hdr_t;

typedef struct pcaprec_hdr_s {
	uint32_t ts_sec;        /* timestamp seconds */
	uint32_t ts_usec;       /* timestamp microseconds */
	uint32_t incl_len;      /* number of octets of packet saved in file */
	uint32_t orig_len;      /* actual length of packet */
} pcaprec_hdr_t;

int main(int argc, char* argv[]) {
	if (argc < 3 || argc > 4) {
		printf("Usage: %s <pci bus id> <output file> [n packets]\n", argv[0]);
		return 1;
	}

	struct ixy_device* dev = ixy_init(argv[1], 1, 1, 0);

	FILE* pcap = fopen(argv[2], "wb");
	if (pcap == NULL) {
		error("failed to open file %s", argv[2]);
	}

	int64_t n_packets = -1;
	if (argc == 4) {
		n_packets = atol(argv[3]);
		printf("Capturing %ld packets...\n", n_packets);
	} else {
		printf("Capturing packets...\n");
	}

	pcap_hdr_t header = {
		.magic_number =  0xa1b2c3d4,
		.version_major = 2,
		.version_minor = 4,
		.thiszone = 0,
		.sigfigs = 0,
		.snaplen = 65535,
		.network = 1, // Ethernet
	};
	fwrite(&header, sizeof(header), 1, pcap);

	struct pkt_buf* bufs[BATCH_SIZE];
	while (n_packets != 0) {
		uint32_t num_rx = ixy_rx_batch(dev, 0, bufs, BATCH_SIZE);
		struct timeval tv;
		gettimeofday(&tv, NULL);

		for (uint32_t i = 0; i < num_rx && n_packets != 0; i++) {
			pcaprec_hdr_t rec_header = {
				.ts_sec = tv.tv_sec,
				.ts_usec = tv.tv_usec,
				.incl_len = bufs[i]->size,
				.orig_len = bufs[i]->size
			};
			fwrite(&rec_header, sizeof(pcaprec_hdr_t), 1, pcap);

			fwrite(bufs[i]->data, bufs[i]->size, 1, pcap);

			pkt_buf_free(bufs[i]);
			// n_packets == -1 indicates unbounded capture
			if (n_packets > 0) {
				n_packets--;
			}
		}
	}

	fclose(pcap);
	return 0;
}