src/interrupts.h

Walkthrough, interview notes & deep dive

This file defines the data structures and constants necessary for ixy's hybrid interrupt-polling model. While low-latency drivers typically busy-poll the NIC to avoid context-switch overhead, this header facilitates a mode where the driver can sleep during periods of low traffic and be woken by hardware interrupts via the VFIO interface.

Constants and Tuning. The macro MOVING_AVERAGE_RANGE defines the window size (5) for calculating packet arrival rates, while INTERRUPT_THRESHOLD (1200) sets the boundary for switching between polling and sleeping. The driver uses these to smooth out traffic bursts and avoid frequent, expensive transitions between modes.

Rate Tracking. The interrupt_moving_avg struct implements a simple sliding window:

struct interrupt_moving_avg {
	uint32_t index;
	uint32_t length;
	uint64_t sum;
	uint64_t measured_rates[MOVING_AVERAGE_RANGE];
};

It tracks the sum and individual measured_rates to provide a stable metric for the check_interrupt logic, which determines if the current packet rate justifies staying in high-power busy-poll mode.

Queue and Device State. The interrupt_queues struct manages per-queue signaling. It contains vfio_event_fd, a file descriptor where the kernel notifies userspace of an interrupt, and vfio_epoll_fd for blocking waits. Notably, instr_counter is used as a lightweight throttle to avoid calling monotonic_time() on every iteration, reducing syscall overhead.

Global Configuration. The interrupts struct aggregates state for the entire device:

struct interrupts {
	bool interrupts_enabled;
	uint32_t itr_rate;
	struct interrupt_queues* queues;
	uint8_t interrupt_type;
	int timeout_ms;
};

Key members include itr_rate for hardware-level interrupt throttling (moderation) and timeout_ms. If timeout_ms is set to -1, the epoll call will block indefinitely until a packet arrives. The interrupt_type tracks whether the device is using standard MSI or the more flexible MSI-X.

The Mechanism. Ixy uses VFIO to map MSI-X vectors to eventfd objects. When the hardware triggers an interrupt, the kernel increments the eventfd counter, waking any thread blocked in epoll_wait. This allows the driver to transition from a "sleeping" state back to active polling only when data is actually present on the wire.

Interview angles

• Poll vs. Interrupt tradeoff: Polling provides the lowest possible latency and zero jitter but consumes 100% CPU. Interrupts save power and CPU cycles during idle time but introduce significant latency (context switches, cache misses) when waking up.
• What is an eventfd? It is a lightweight kernel-level counter used for signaling. In VFIO drivers, it bridges hardware interrupts to userspace, allowing standard tools like epoll or select to wait on hardware events.
• Interrupt Moderation/Throttling: This is a hardware feature (controlled by itr_rate) that prevents "interrupt storms" by ensuring the NIC doesn't fire an interrupt for every single packet, instead waiting for a timer or a batch of packets to minimize CPU disruption.

Going deeper

struct interrupt_moving_avg {
	uint32_t index;
	uint32_t length;
	uint64_t sum;
	uint64_t measured_rates[MOVING_AVERAGE_RANGE];
};

The sum is uint64_t to prevent overflow when aggregating high-frequency packet rates. The index advances modulo MOVING_AVERAGE_RANGE (5) to implement a circular buffer. Efficiently updating the average requires an incremental "subtract-old-add-new" approach rather than rescanning the array. Note that length tracks the warmup period; until it reaches the window size, the divisor for the average must be length, not the constant range.

struct interrupts {
	bool interrupts_enabled;
	uint32_t itr_rate;
	struct interrupt_queues* queues;
	int timeout_ms;
};

The timeout_ms is a signed int because it acts as a sentinel for epoll_wait: -1 triggers an infinite block, while positive values enable periodic wakeups. The queues pointer targets an array of struct interrupt_queues, allowing the driver to scale based on the number of hardware RX/TX rings initialized.

Harder interview questions

• Can the uint64_t sum or rx_pkts overflow? Practically no: at line rate a 64-bit counter takes centuries to wrap, so the driver omits overflow checks. But the moving-average sum is bounded differently — it only holds the last 5 samples, so its real risk is a stale or garbage slot, not arithmetic wraparound.
• What are the concurrency implications of this header? The structures are designed for a 1:1 mapping between CPU cores and hardware queues (Siloing). There are no mutexes or atomics; if two threads access the same queue state, data races on rx_pkts and sum will lead to undefined behavior and incorrect throttling decisions.
• How does the moving average handle bursts? Since the window is small (MOVING_AVERAGE_RANGE = 5), the driver reacts quickly to traffic spikes. However, without a hysteresis mechanism, the system might "flap" between polling and interrupts if the rate oscillates around the 1200 threshold, causing significant jitter.

Gotchas

• The interrupt_queues struct is not cache-line aligned. In multi-queue setups, the state for Queue 0 and Queue 1 may share a cache line, triggering false sharing performance hits if different cores manage them.
• If measured_rates is not explicitly zeroed during allocation, the incremental sum update will incorporate garbage memory, resulting in nonsensical interrupt-moderation decisions until the first 5 samples are processed.

From ixy to a production driver

In a production Linux driver such as ixgbe.ko, this RX interrupt problem is handled by NAPI, not by a userspace moving average. The NIC raises an interrupt, ixgbe masks further queue interrupts, schedules NAPI, and the kernel later calls ixgbe_poll from softirq context. That poll loop drains RX work up to the NAPI budget; the default driver weight is commonly 64 packets. Interrupts are re-enabled only when the ring is no longer busy. This is the classic transition: interrupts wake an idle system, polling handles bursts without one interrupt per packet.

ixy's interrupts.h models the same tradeoff in a smaller way. It keeps per-queue counters, a 5-sample moving average, and an INTERRUPT_THRESHOLD of 1200 packets per interval. Below the threshold it arms MSI/MSI-X through VFIO, blocks in epoll_wait, and lets the kernel bump an eventfd. Above the threshold it keeps polling. This is coarser than NAPI: no softirq scheduling, per-NAPI state machine, fairness across devices, or kernel-owned re-scheduling rules.

Production ixgbe also leans harder on hardware. Intel 82599 has per-vector EITR, the Extended Interrupt Throttle Register, so ixgbe can moderate interrupt frequency. Linux exposes coalescing through ethtool -C, and ixgbe supports adaptive moderation at runtime. ixy exposes only a static itr_rate; it omits the feedback loop, statistics, and operational knobs expected on a real host.

The scaling story is different too. ixgbe allocates MSI-X vectors per queue, uses RSS to spread flows, and relies on IRQ affinity, smp_affinity, or irqbalance so a queue's interrupt lands near its polling CPU. The kernel must be correct under SMP, IRQ context, softirq context, NAPI completion races, reset, hotplug, and ethtool reconfiguration. ixy assumes one core owns one queue, so rx_pkts and sum do not need atomics or locks.

Virtio-net solves the same batching problem at the virtqueue layer. It can suppress notifications with VRING_AVAIL_F_NO_INTERRUPT, VRING_USED_F_NO_NOTIFY, or, with VIRTIO_RING_F_EVENT_IDX, the used-event and avail-event indices. That lets the guest and device say "notify me when the ring reaches this point," reducing VM exits and interrupt churn.

DPDK starts from the opposite default: poll-mode drivers normally avoid RX interrupts for throughput and latency predictability. But DPDK also has RX interrupt mode, rte_eth_dev_rx_intr_*, Linux EAL wakeups based on VFIO/UIO file descriptors and epoll, and rte_epoll_wait. That is the closest production analogue to ixy's userspace design. Interviewers want the tradeoff: interrupts save power when idle, polling avoids interrupt storms, moderation trades latency for CPU, and userspace uses eventfd/epoll because the real ISR remains in the kernel.

Sources

• NAPI
• Intel 82599 Datasheet
• DPDK EAL
• Virtual I/O Device (VIRTIO) Version 1.0

Source

#ifndef IXY_INTERRUPTS_H
#define IXY_INTERRUPTS_H

#include <stdint.h>
#include <stddef.h>
#include <time.h>
#include <stdbool.h>

#define MOVING_AVERAGE_RANGE 5
#define INTERRUPT_THRESHOLD 1200

struct interrupt_moving_avg {
	uint32_t index; // The current index
	uint32_t length; // The moving average length
	uint64_t sum; // The moving average sum
	uint64_t measured_rates[MOVING_AVERAGE_RANGE]; // The moving average window
};

struct interrupt_queues {
	int vfio_event_fd; // event fd
	int vfio_epoll_fd; // epoll fd
	bool interrupt_enabled; // Whether interrupt for this queue is enabled or not
	uint64_t last_time_checked; // Last time the interrupt flag was checked
	uint64_t instr_counter; // Instruction counter to avoid unnecessary calls to monotonic_time
	uint64_t rx_pkts; // The number of received packets since the last check
	uint64_t interval; // The interval to check the interrupt flag
	struct interrupt_moving_avg moving_avg; // The moving average of the hybrid interrupt
};

struct interrupts {
	bool interrupts_enabled; // Whether interrupts for this device are enabled or disabled.
	uint32_t itr_rate; // The Interrupt Throttling Rate
	struct interrupt_queues* queues; // Interrupt settings per queue
	uint8_t interrupt_type; // MSI or MSIX
	int timeout_ms; // interrupt timeout in milliseconds (-1 to disable the timeout)
};

void check_interrupt(struct interrupt_queues* interrupt, uint64_t diff, uint32_t buf_index, uint32_t buf_size);

#endif //IXY_INTERRUPTS_H