src/interrupts.c

Walkthrough, interview notes & deep dive

What this file does: The interrupts.c file implements the hybrid interrupt-and-polling logic for the ixy driver. In high-performance networking, purely interrupt-driven I/O is often avoided due to the high cost of context switching and interrupt latency. Conversely, pure busy-polling consumes 100% of a CPU core even when there is no traffic. This source file provides the mechanism to dynamically switch between these modes based on traffic volume, allowing the driver to yield the CPU during idle periods while maintaining high throughput during bursts.

The mechanism: To handle hardware interrupts in userspace, ixy utilizes the Linux VFIO (Virtual Function I/O) framework. When a NIC triggers an MSI-X interrupt, the kernel-side VFIO driver catches it and signals a corresponding eventfd held by the userspace application. The driver can then use standard Linux system calls like epoll to sleep until data arrives. This approach bridges the gap between low-level hardware signaling and userspace application logic without the overhead of a custom kernel module.

Function walk-through: The file begins with a static helper function called ppms(). This function calculates the packets-per-millisecond rate by taking the received_pkts count and dividing it by the elapsed_time_nanos (normalized to milliseconds). This rate provides a standardized metric for traffic intensity regardless of how frequently the check logic is invoked.

static uint64_t ppms(uint64_t received_pkts, uint64_t elapsed_time_nanos) {
	return received_pkts / (elapsed_time_nanos / 1000000);
}

Moving average logic: The core of the decision-making process resides in check_interrupt(). This function manages a struct interrupt_moving_avg within the interrupt_queues structure. It uses a sliding window approach where the sum of recent rates is updated by subtracting the oldest value in measured_rates and adding the latest calculation. The index is incremented modulo MOVING_AVERAGE_RANGE to maintain the ring-buffer structure, and the length tracks how many samples have been collected before the window is fully populated.

struct interrupt_moving_avg* avg = &interrupt->moving_avg;
avg->sum -= avg->measured_rates[avg->index];
avg->measured_rates[avg->index] = ppms(interrupt->rx_pkts, diff);
avg->sum += avg->measured_rates[avg->index];
if (avg->length < MOVING_AVERAGE_RANGE) {
	avg->length++;
}
avg->index = (avg->index + 1) % MOVING_AVERAGE_RANGE;

Decision logic and thresholds: Once the average rate is calculated, the driver compares it against the INTERRUPT_THRESHOLD. If the traffic is high, interrupt_enabled is set to false, committing the driver to busy-polling. This prevents an "interrupt storm" where the CPU spends more time handling context switches than processing packets. Additionally, the function checks if buf_index equals buf_size. A full buffer is a critical signal that the system is under heavy load, requiring polling to drain the RX ring as fast as possible.

uint64_t average = avg->sum / avg->length;
if (average > INTERRUPT_THRESHOLD) {
	interrupt->interrupt_enabled = false;
} else if (buf_index == buf_size) {
	interrupt->interrupt_enabled = false;
} else {
	interrupt->interrupt_enabled = true;
}
interrupt->last_time_checked = monotonic_time();

State management: After each check, the rx_pkts counter is reset to zero to begin the next measurement period. The last_time_checked is updated using monotonic_time(), which ensures that the time delta (diff) used in subsequent calls is accurate and unaffected by system clock jumps. This smoothed, time-aware approach ensures the driver doesn't flap between modes too rapidly, which would itself introduce significant latency overhead.

Interview angles

• Poll vs. Interrupt tradeoff: Polling offers the lowest latency and highest throughput under load but wastes power and CPU cycles when idle. Interrupts save power but introduce context-switch overhead and "livelock" risks where the system spends all its time in the ISR. Hybrid models like ixy’s attempt to find the Pareto optimal point by switching modes based on traffic density.
• What is an eventfd: An eventfd is a Linux-specific file descriptor used for event notification. In this context, it allows the kernel (via VFIO) to notify userspace that a hardware interrupt has occurred. It is lightweight because it essentially acts as a 64-bit counter in kernel memory that can be "read" or "written" to signal events, making it compatible with select, poll, and epoll.
• Interrupt moderation/throttling: Hardware NICs often support "Interrupt Coalescing," where the hardware waits for a certain number of packets or a timeout before firing an interrupt. The check_interrupt logic in interrupts.c is essentially a software-level implementation of this concept, deciding when the "cost" of the interrupt is justified by the "work" available to be done.

Going deeper

static uint64_t ppms(uint64_t received_pkts, uint64_t elapsed_time_nanos) {
	return received_pkts / (elapsed_time_nanos / 1000000);
}

The ppms calculation contains a critical arithmetic risk. Because it uses integer division for elapsed_time_nanos / 1000000, any interval shorter than 1ms results in a divisor of zero, triggering a hardware exception (SIGFPE) that will crash the driver. Furthermore, integer truncation means a rate of 1999 packets per 2ms reflects as 999 pkts/ms, losing precision that might be relevant for tight moderation thresholds.

avg->sum -= avg->measured_rates[avg->index];
avg->measured_rates[avg->index] = ppms(interrupt->rx_pkts, diff);
avg->sum += avg->measured_rates[avg->index];
if (avg->length < MOVING_AVERAGE_RANGE) {
	avg->length++;
}
avg->index = (avg->index + 1) % MOVING_AVERAGE_RANGE;

The moving average update uses a subtract-then-add pattern to maintain avg->sum in O(1) time. By subtracting the "stale" value at the current index before overwriting it with the new rate, the sum remains perfectly synchronized with the window content without re-summing the entire array. The length counter is incremented before the index advances but clamped to MOVING_AVERAGE_RANGE, ensuring that the average = sum / length calculation remains mathematically sound even during the "warm-up" phase before the ring buffer is full.

if (average > INTERRUPT_THRESHOLD) {
	interrupt->interrupt_enabled = false;
} else if (buf_index == buf_size) {
	interrupt->interrupt_enabled = false;
} else {
	interrupt->interrupt_enabled = true;
}

The decision logic introduces a secondary override: buf_index == buf_size. While the primary mode switch depends on the packet rate (average), a full descriptor ring forces interrupt_enabled = false (polling mode) regardless of the rate. This acts as a pressure valve: if the hardware cannot offload packets because the software is too slow, the system must stay in the active polling loop to drain the ring and prevent a deadlock where no new packets can arrive to trigger a rate-based mode change.

Harder interview questions

• Q: Why use a moving average instead of an instantaneous rate check? A: To provide hysteresis and prevent "mode flapping." An instantaneous burst could disable interrupts only for the rate to drop 100μs later, forcing the system into a costly cycle of enabling/disabling IRQs and context switching.
• Q: Is check_interrupt thread-safe? A: No. It assumes a single-threaded driver model. If rx_pkts is incremented by a RX thread while check_interrupt is called by a management thread, the interrupt->rx_pkts = 0 reset is a classic race condition that leads to dropped counts or double-counting.
• Q: How could you optimize the modulo operator in the index update? A: index % MOVING_AVERAGE_RANGE is an expensive division instruction. A senior engineer would define MOVING_AVERAGE_RANGE as a power of two (e.g., 64) and use a bitwise mask (index = (index + 1) & 63), which executes in a single clock cycle.
• Q: Why use monotonic_time() instead of wall-clock time? A: Wall-clock time (gettimeofday) can jump backward or forward due to NTP updates or leap seconds. monotonic_time() is guaranteed to be non-decreasing, which is essential for calculating a reliable diff for rate measurements.

Gotchas

• Divide-by-zero: The driver will crash if check_interrupt is invoked twice within the same millisecond due to the elapsed_time_nanos / 1000000 truncation.
• Uninitialized sum: If the interrupt_queues structure is not zero-initialized (e.g. allocated via malloc instead of calloc), the avg->sum will start with garbage data, resulting in incorrect polling decisions for the first MOVING_AVERAGE_RANGE cycles.
• Integer Truncation: Very low rates (e.g., 1 packet every 2ms) will be truncated to 0 pkts/ms, potentially leaving the NIC in interrupt mode when it should be polling if the per-packet interrupt overhead is extremely high.

From ixy to a production driver

Kernel-side pattern In Linux ixgbe, the same job is split between the NIC, the hard IRQ handler, and NAPI. A receive interrupt arrives through an MSI-X vector, the driver masks further device interrupts with registers such as EIMC, schedules a NAPI poll instance, and the networking softirq path (net_rx_action) calls the driver's poll routine up to a budget. Only when napi_poll drains the ring and comes up empty does the driver complete NAPI and re-enable interrupts, often through EIMS. That is the classic receive-livelock answer from Mogul and Ramakrishnan: interrupt as entry signal, polling to drain work. ixy flips the default. Its app loop is poll-first, and interrupts.c only decides whether the next idle period may sleep on a VFIO eventfd.

Silicon assist The 82599 already has interrupt moderation in hardware. EITR (Extended Interrupt Throttle Registers) implements an ITR, enforcing a minimum interval between interrupts at microsecond scale, and production ixgbe exposes coalescing policy through ethtool -C knobs such as rx-usecs and adaptive receive moderation (adaptive-rx). ixy deliberately does not program EITR; its struct interrupt_moving_avg plus INTERRUPT_THRESHOLD is a coarse software imitation. That is useful for teaching because the policy is visible in one file, but production drivers want hardware pacing because coalescing is a latency/throughput/CPU tradeoff.

Dataplane contrast DPDK PMDs are closer to ixy philosophically: they default to busy-polling descriptor rings because high packet rates are faster without IRQ delivery. Interrupts are an opt-in power-saving path, exposed through APIs such as rte_eth_dev_rx_intr_ctl, rte_eth_dev_rx_intr_enable, and examples like l3fwd-power, where a core sleeps on an epoll-backed RX interrupt only after finding no packets. ixy demonstrates that same hybrid idea, but without DPDK's broader device support, error paths, and queue-level integration.

Production gaps to name in interview ixy omits EITR programming, ethtool -C controls, ethtool stats, hotplug, retry/error handling on the eventfd path, and real power-management transitions into lower C-states. It is single-queue and single-threaded, so it avoids multiqueue RSS, per-queue MSI-X vector steering, and IRQ affinity; those are essential when scaling across cores. It also has no locking around the rx_pkts reset, so a multithreaded version would race, and ppms() has no divide-by-zero or integer-overflow hardening. For an interviewer, each omission maps to a concept: why NAPI prevents receive livelock, why coalescing trades latency for CPU efficiency, why RSS plus IRQ affinity matters, and why busy-polling wins at load while interrupts still matter for idle power.

Sources

• NAPI - The Linux Kernel documentation
• Intel 82599 10 Gigabit Ethernet Controller Datasheet
• DPDK L3 Forwarding with Power Management Sample Application
• User Space Network Drivers

Source

#include "interrupts.h"
#include "libixy-vfio.h"
#include "log.h"
#include "stats.h"

#include <stdio.h>

/**
 * Calculate packets per millisecond based on the received number of packets and the elapsed time in nanoseconds since the
 * last calculation.
 * @param received_pkts Number of received packets.
 * @param elapsed_time_nanos Time elapsed in nanoseconds since the last calculation.
 * @return Packets per millisecond.
 */
static uint64_t ppms(uint64_t received_pkts, uint64_t elapsed_time_nanos) {
	return received_pkts / (elapsed_time_nanos / 1000000);
}

/**
 * Check if interrupts or polling should be used based on the current number of received packets per seconds.
 * @param interrupts The interrupt handler.
 * @param diff The difference since the last call in nanoseconds.
 * @param buf_index The current buffer index.
 * @param buf_size The maximum buffer size.
 * @return Whether to disable NIC interrupts or not.
 */
void check_interrupt(struct interrupt_queues* interrupt, uint64_t diff, uint32_t buf_index, uint32_t buf_size) {
	struct interrupt_moving_avg* avg = &interrupt->moving_avg;
	avg->sum -= avg->measured_rates[avg->index];
	avg->measured_rates[avg->index] = ppms(interrupt->rx_pkts, diff);
	avg->sum += avg->measured_rates[avg->index];
	if (avg->length < MOVING_AVERAGE_RANGE) {
		avg->length++;
	}
	avg->index = (avg->index + 1) % MOVING_AVERAGE_RANGE;
	interrupt->rx_pkts = 0;
	uint64_t average = avg->sum / avg->length;
	if (average > INTERRUPT_THRESHOLD) {
		interrupt->interrupt_enabled = false;
	} else if (buf_index == buf_size) {
		interrupt->interrupt_enabled = false;
	} else {
		interrupt->interrupt_enabled = true;
	}
	interrupt->last_time_checked = monotonic_time();
}