src/stats.c

Walkthrough, interview notes & deep dive

stats.c provides the telemetry and performance monitoring infrastructure for the ixy userspace driver. It is responsible for tracking cumulative traffic metrics and calculating real-time throughput. For an engineer working on low-latency NICs like Solarflare, understanding how these statistics are derived is crucial for debugging the datapath and verifying that the system is hitting "wire rate" performance.

The implementation centers around struct device_stats, which stores rx_pkts, tx_pkts, rx_bytes, and tx_bytes. The lifecycle begins with stats_init, which clears these counters and calls ixy_read_stats to reset the hardware registers on the NIC. High-precision timing is provided by monotonic_time, which uses clock_gettime(CLOCK_MONOTONIC) to return a nanosecond-resolution timestamp.

The core logic for performance analysis is found in the rate-calculation functions. diff_mpps calculates millions of packets per second, while diff_mbit computes the bit rate. A critical detail in diff_mbit is the handling of Ethernet overhead:

static uint32_t diff_mbit(uint64_t bytes_new, uint64_t bytes_old, uint64_t pkts_new, uint64_t pkts_old, uint64_t nanos) {
	return (uint32_t) (((bytes_new - bytes_old) / 1000000.0 / ((double) nanos / 1000000000.0)) * 8
		+ diff_mpps(pkts_new, pkts_old, nanos) * 20 * 8);
}

The addition of diff_mpps(...) * 20 * 8 is a standard networking practice to account for the physical layer (L1) overhead: the 7-byte Preamble, the 1-byte Start Frame Delimiter (SFD), and the 12-byte Inter-Frame Gap (IFG). Without this 20-byte-per-packet correction, a 10GbE link saturated with minimum-sized 64-byte packets would only appear to be running at ~7.6 Gbps. Including this overhead allows the print_stats_diff function to correctly report a 10000 Mbit/s line rate when the link is fully utilized.

This file demonstrates the importance of separating raw hardware counters (which usually only track the L2 Ethernet frame) from reported bandwidth metrics that must reflect the physical capacity of the wire.

Interview angles

Q: Why does the diff_mbit calculation add 20 bytes to every packet? A: Hardware counters typically only record the Ethernet L2 frame size (destination MAC through CRC). To accurately report the physical line rate (e.g., 10Gbps), you must add the L1 overhead consisting of the Preamble, SFD, and Inter-Frame Gap.

Q: Why use CLOCK_MONOTONIC instead of CLOCK_REALTIME for monotonic_time? A: CLOCK_MONOTONIC represents absolute elapsed time and is not affected by system clock jumps or NTP adjustments. This is essential for calculating accurate deltas between samples in performance monitoring.

Q: What is the performance trade-off of calling monotonic_time or ixy_read_stats in the main loop? A: These calls involve syscalls or MMIO reads which are expensive. In a production low-latency driver, statistics are usually gathered out-of-band or sampled infrequently (e.g., every 1 second) to avoid polluting the instruction cache and adding latency to the fast path.

Going deeper

uint64_t monotonic_time() {
	struct timespec timespec;
	clock_gettime(CLOCK_MONOTONIC, &timespec);
	return timespec.tv_sec * 1000 * 1000 * 1000 + timespec.tv_nsec;
}

Relies on tv_sec being 64-bit (standard on modern Linux). If this were a 32-bit time_t system, the multiplication tv_sec * 10^9 would overflow in just 4.2 seconds. Using uint64_t ensures the final timestamp is stable for centuries, and unsigned subtraction for deltas handles the 64-bit wrap-around gracefully.

void stats_init(struct device_stats* stats, struct ixy_device* dev) {
	...
	if (dev) {
		ixy_read_stats(dev, NULL);
	}
}

The dummy read to ixy_read_stats with a NULL pointer is critical for hardware synchronization. Many NIC registers are "clear-on-read." By reading and discarding the current hardware values during initialization, the driver prevents "ghost" packets (accumulated before the driver started) from creating a massive artificial spike in the first reported rate.

Harder interview questions

• Is the lack of locking in device_stats a thread-safety bug? In ixy, the poll-mode thread usually owns the stats. If a separate monitoring thread reads them, it performs an unsynchronized 64-bit load. On x86-64, aligned 64-bit operations are atomic, so you won't see "torn" values (half-old, half-new), but you might see slightly stale data without an explicit memory barrier.
• How does bytes_new - bytes_old handle hardware counter wrap-around? Since both are uint64_t, standard unsigned power-of-two modular arithmetic applies. As long as the counter hasn't wrapped twice between samples, the subtraction correctly yields the positive delta even if the "new" value is numerically smaller than the "old" one.

Gotchas

• External Interference: Since hardware counters often clear on read, running external tools like ethtool -S or tcpdump simultaneously can "steal" counts from the driver, leading to inaccurate or zeroed stats in your application.
• Truncation Risk: diff_mbit performs calculations in double but casts the final result to uint32_t. While safe for Mbit/s on 10GbE links, this would overflow if reporting raw bits/s on 100GbE+ hardware.

From ixy to a production driver

Hardware truth: on an 82599, the real counters are not the four fields in struct device_stats. The NIC exposes a broad MMIO statistics block: GPRC/GPTC for good packets, GORCL/GORCH and GOTCL/GOTCH for 64-bit good-octet counters split into low/high registers, QPRC/QPTC per-queue packet counters, and loss/error counters such as MPC, RNBC, and CRCERRS. ixy's 82599 path reads only the global good packet/octet subset and discards the rest. That is fine for a teaching throughput display, but it means "RX packets went down" cannot be separated from RSS imbalance, no-buffer loss, missed packets, CRC failures, or queue-local starvation.

Linux model: ixgbe.ko treats statistics as driver state, not just a print helper. Its service/watchdog path folds hardware reads into struct ixgbe_hw_stats and netdev-facing counters such as struct rtnl_link_stats64; ethtool -S then exposes a much larger xstats-style set, including named NIC totals, error counters, flow-control counters, RSC counters, and per-queue rx_queue_%u_* / tx_queue_%u_* values. One low-level detail interviewers care about is 64-bit MMIO composition: split byte counters need a defined order, with the high half read after the low half for the latched value or clear behavior. stats.c has no such hardware contract; the small ixy 82599 reader happens to use low-then-high for its two byte counters, but it does not generalize that discipline across the full register bank.

DPDK model: DPDK draws the same line in API form. rte_eth_stats_get() returns the basic rte_eth_stats structure, including packet, byte, error, missed, and no-mbuf fields when the PMD supports them. rte_eth_xstats_get() is the extended path for driver/NIC-specific counters and per-queue detail. The PMD owns the messy parts: hardware wrap, aggregation across queues or pools, and software fallback counters for things the device cannot count directly.

What ixy omits on purpose: there is no per-queue accounting, no drop/error/CRC reporting, no full 64-bit hi/lo latch policy, no locking for multiple stats readers, no netlink or ethtool integration, and no offload-aware interpretation. With TSO, RSC/LRO, checksum offload, or descriptor coalescing, "a packet" at the OS, descriptor, and wire levels may not be the same unit.

AMD/Solarflare angle: the expected production-driver instinct is diagnostic. If packets disappear, first check MPC/RNBC/no-buffer style drops before blaming the application. If one core is hot, look at per-queue/RSS counters before changing ring size. If a 64-bit byte counter is split into 32-bit registers, ask what latches what, in which order, and whether reads clear state. ixy intentionally hides those concerns so the source stays small enough to teach.

Sources

• Intel 82599 10 Gigabit Ethernet Controller Datasheet
• Linux ixgbe driver source
• DPDK Ethernet Device API
• User Space Network Drivers

Source

#include "stats.h"

#include <stdio.h>

void print_stats(struct device_stats* stats) {
	printf("[%s] RX: %zu bytes %zu packets\n", stats->device ? stats->device->pci_addr : "???", stats->rx_bytes, stats->rx_pkts);
	printf("[%s] TX: %zu bytes %zu packets\n", stats->device ? stats->device->pci_addr : "???", stats->tx_bytes, stats->tx_pkts);
}

static double diff_mpps(uint64_t pkts_new, uint64_t pkts_old, uint64_t nanos) {
	return (double) (pkts_new - pkts_old) / 1000000.0 / ((double) nanos / 1000000000.0);
}

static uint32_t diff_mbit(uint64_t bytes_new, uint64_t bytes_old, uint64_t pkts_new, uint64_t pkts_old, uint64_t nanos) {
	// take stuff on the wire into account, i.e., the preamble, SFD and IFG (20 bytes)
	// otherwise it won't show up as 10000 mbit/s with small packets which is confusing
	return (uint32_t) (((bytes_new - bytes_old) / 1000000.0 / ((double) nanos / 1000000000.0)) * 8
		+ diff_mpps(pkts_new, pkts_old, nanos) * 20 * 8);
}

void print_stats_diff(struct device_stats* stats_new, struct device_stats* stats_old, uint64_t nanos) {
	printf("[%s] RX: %d Mbit/s %.2f Mpps\n", stats_new->device ? stats_new->device->pci_addr : "???",
		diff_mbit(stats_new->rx_bytes, stats_old->rx_bytes, stats_new->rx_pkts, stats_old->rx_pkts, nanos),
		diff_mpps(stats_new->rx_pkts, stats_old->rx_pkts, nanos)
	);
	printf("[%s] TX: %d Mbit/s %.2f Mpps\n", stats_new->device ? stats_new->device->pci_addr : "???",
		diff_mbit(stats_new->tx_bytes, stats_old->tx_bytes, stats_new->tx_pkts, stats_old->tx_pkts, nanos),
		diff_mpps(stats_new->tx_pkts, stats_old->tx_pkts, nanos)
	);
}


// returns a timestamp in nanoseconds
// based on rdtsc on reasonably configured systems and is hence fast
uint64_t monotonic_time() {
	struct timespec timespec;
	clock_gettime(CLOCK_MONOTONIC, &timespec);
	return timespec.tv_sec * 1000 * 1000 * 1000 + timespec.tv_nsec;
}

// initializes a stat struct and clears the stats on the device
void stats_init(struct device_stats* stats, struct ixy_device* dev) {
	// might require device-specific initialization
	stats->rx_pkts = 0;
	stats->tx_pkts = 0;
	stats->rx_bytes = 0;
	stats->tx_bytes = 0;
	stats->device = dev;
	if (dev) {
		ixy_read_stats(dev, NULL);
	}
}