src/driver/virtio_type.h

virtio-net register and virtqueue definitions.

Walkthrough, interview notes & deep dive

The Hardware Interface and Handshake. This header defines the register map and memory structures for VirtIO devices operating over a legacy PCI bus. It captures the offsets found in the Base Address Register (BAR) 0, which act as the primary control interface. Registers like VIRTIO_PCI_QUEUE_SEL allow the driver to select a specific Virtqueue, after which it can read its size from VIRTIO_PCI_QUEUE_NUM and write its physical address to VIRTIO_PCI_QUEUE_PFN. The status register, VIRTIO_PCI_STATUS, facilitates a state-machine handshake between the guest and host using bits like VIRTIO_CONFIG_STATUS_ACK and VIRTIO_CONFIG_STATUS_DRIVER_OK.

The Virtqueue Split-Ring Architecture. The core data path relies on a shared-memory mechanism called a Virtqueue. This header defines the three constituent structures that reside in DMA-accessible memory: the Descriptor Table, the Available Ring, and the Used Ring. The struct vring_desc defines the descriptors that point to packet data, while vring_avail is used by the guest to signal new buffers to the device. Conversely, the device uses vring_used and vring_used_elem to report completed operations back to the driver.

struct vring_desc {
	uint64_t addr;  /*  Address (guest-physical). */
	uint32_t len;   /* Length. */
	uint16_t flags; /* The flags as indicated above. */
	uint16_t next;  /* We chain unused descriptors via this. */
};

Packet Metadata and Offloading. For networking specifically, the virtio_legacy_net_hdr struct provides per-packet metadata. This is prepended to every transmitted and received frame. It allows the driver and hardware to negotiate complex features such as checksum offloading via VIRTIO_NET_HDR_F_NEEDS_CSUM and Segmentation Offload via gso_type. Feature bits like VIRTIO_NET_F_MRG_RXBUF enable the host to merge multiple receive buffers for large packets, which is critical for high-throughput performance.

Mechanism Coordination. The driver coordinates with the device by populating descriptors and updating the vring_avail->idx. It then triggers a "kick" by writing the queue index to the VIRTIO_PCI_QUEUE_NOTIFY register. The device processes the descriptors and updates vring_used->idx. To minimize overhead, the header includes flags like VRING_USED_F_NO_NOTIFY and VRING_AVAIL_F_NO_INTERRUPT to suppress unnecessary notifications when the other side is already polling.

Interview angles

Q: How does the legacy VirtIO interface handle memory addresses on 64-bit systems?
A: Even though VIRTIO_PCI_QUEUE_PFN is a 32-bit register, it represents a Page Frame Number. By applying the VIRTIO_PCI_QUEUE_ADDR_SHIFT of 12 (representing 4KB pages), the device can address physical memory up to 44 bits, which is sufficient for many server architectures.

Q: What is the significance of the `VRING_DESC_F_INDIRECT` flag?
A: It allows a single descriptor to point to a separate table of descriptors. This significantly increases the maximum number of buffers that can be submitted in a single notification, reducing the pressure on the primary ring and improving cache efficiency for large scatter-gather operations.

Going deeper

Distance math and wraparound

static inline int vring_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old) {
	return (uint16_t)(new_idx - event_idx - 1) < (uint16_t)(new_idx - old);
}

This function solves the "has the device passed my threshold?" problem for free-running 16-bit counters that wrap at 65535. By casting the results of the subtractions to uint16_t, it calculates the distance between indices in circular space. The condition checks if the distance from the threshold event_idx to the current new_idx is smaller than the distance the index just traveled (new_idx - old). This ensures a notification is triggered exactly once when the counter crosses the threshold, even if it wraps around zero.

Compiler vs. Hardware ordering

struct vring_used {
	uint16_t flags;
	volatile uint16_t idx;
	struct vring_used_elem ring[0];
};

The volatile keyword on idx is a directive to the compiler to prevent it from caching the value in a CPU register. This is mandatory because the device (Host) updates this memory asynchronously. However, for a senior interview, remember that volatile is not a memory barrier. On weakly-ordered architectures like ARM, the CPU might reorder the read of the data in the ring array to occur before the read of the idx "doorbell." A true system requires an architectural memory barrier (like rmb()) to ensure the host has actually finished writing the descriptor data before the guest processes it.

The phantom index layout

/*
 * We publish the used event index at the end of the available ring, and vice
 * versa. They are at the end for backwards compatibility.
 */
#define vring_used_event(vr) ((vr)->avail->ring[(vr)->num])
#define vring_avail_event(vr) (*(uint16_t*)&(vr)->used->ring[(vr)->num])

These macros exploit the memory layout of the rings. The ring[0] syntax defines a flexible array member. While the standard ring has num entries, the driver allocates enough memory for num + 1. The "extra" slot at the very end is used to store the event suppression index for the other side. This clever packing allows newer drivers to support fine-grained event notification without breaking the binary structure of the legacy vring_avail and vring_used headers.

Harder interview questions

Q: Why does the `vring` struct use a power-of-two for `num`? A: It allows the driver to use a bitwise AND (idx & (num - 1)) to calculate the array index from the free-running 16-bit counter, which is significantly faster than a modulo operation in the fast path.
Q: What is the risk of a high `VIRTIO_PCI_QUEUE_ADDR_SHIFT`? A: This shift (fixed at 12 for 4KB pages) means the QUEUE_PFN register can only address memory at 4KB alignments. If the system uses a different page size or requires sub-page alignment for DMA regions, the driver must carefully pad the allocation to ensure the base address is a multiple of 4096.
Q: How does `VRING_DESC_F_INDIRECT` impact cache performance? A: It allows batching many buffers into one "master" descriptor, but it introduces an extra pointer dereference (pointer chasing). For small packets, the overhead of fetching the indirect table from memory can outweigh the benefit of reduced ring pressure.
Q: If the Host is stuck, how can the Guest detect a ring hang using only this header? A: The guest can monitor the used->idx. If the guest has placed descriptors in the avail ring but used->idx does not progress over a timeout period while the device status remains DRIVER_OK, a device-level reset (writing 0 to VIRTIO_PCI_STATUS) is likely required.

Gotchas

The vring_avail_event macro performs a pointer cast *(uint16_t*) on a struct vring_used_elem array. Since the element is 8 bytes and the event index is 2 bytes, this relies on the host and guest agreeing on the exact alignment of that phantom slot at the end of the ring.
In vring_desc, the addr is a 64-bit guest-physical address. If the driver is running in a VM with an IOMMU, this must be the IOVA (I/O Virtual Address), not the raw physical address of the buffer.

From ixy to a production driver

Transport generalization In virtio_type.h, ixy freezes the old virtio PCI transport into constants: BAR0 I/O offsets such as VIRTIO_PCI_QUEUE_PFN, VIRTIO_PCI_QUEUE_NOTIFY, VIRTIO_PCI_STATUS, and VIRTIO_PCI_ISR. A production driver cannot assume that layout. Linux's PCI path has a modern-device side around virtio_pci_modern_dev, virtio_pci_modern.c, and virtio_config_ops; it discovers VIRTIO_PCI_CAP_COMMON_CFG, notify, ISR, and device configuration areas from PCI capabilities. DPDK's virtio PMD similarly supports modern virtio 1.0+ devices. With VIRTIO_F_VERSION_1, feature negotiation is not just "write a mask and continue": the driver must set FEATURES_OK, reread status, and fail cleanly if the device rejects the set.

Queue model in practice This header defines only the classic split vring: vring_desc, vring_avail, vring_used, and the event-index helper vring_need_event. Linux's virtqueue layer hides PCI versus MMIO and supports both split rings and packed rings when VIRTIO_F_RING_PACKED is negotiated. The interviewer-level point is that ixy teaches the memory layout directly, while Linux turns it into a transport-independent contract with DMA mapping, barriers, callbacks, and ring variants behind helpers.

Runtime integration Real virtio_net is part of the network stack. It wires receive processing through NAPI paths such as virtnet_poll, uses MSI-X queue/config vectors, and can switch between interrupt-driven wakeups and polling. ixy is deliberately pure polling: it may name VIRTIO_PCI_ISR and event suppression fields, but it does not build the power/latency machinery around interrupts, NAPI budgets, or adaptive moderation.

Feature depth Production virtio-net also drives features that this header mostly names: VIRTIO_NET_F_MQ plus VIRTIO_NET_F_CTRL_VQ for multiqueue setup and steering, control commands for RX mode/MAC/VLAN, checksum and TSO/GSO offloads through the kernel skb or DPDK mbuf metadata, and VIRTIO_NET_F_MRG_RXBUF for receive buffer chaining. ixy instead uses one queue pair and a minimal virtio_legacy_net_hdr, because the educational goal is to expose descriptor ownership and packet movement without burying them under stack integration.

Production obligations The biggest hidden gap is robustness. A real driver must honor VIRTIO_F_IOMMU_PLATFORM/VIRTIO_F_ACCESS_PLATFORM by using DMA-API IOVAs, while legacy ixy writes a 32-bit page frame number to QUEUE_PFN with VIRTIO_PCI_QUEUE_ADDR_SHIFT. It must also handle reset after DRIVER_FAILED, hot removal, retries, stats, locking around concurrent TX, and partial feature fallback. In an interview, call out that ixy omits these not because they are optional in production, but because each would obscure the core low-level mechanism this header is meant to make visible.

Sources

Source

filesrc/driver/virtio_type.h

/*-
 *   BSD LICENSE
 *
 *   Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
 *   All rights reserved.
 *
 *   Redistribution and use in source and binary forms, with or without
 *   modification, are permitted provided that the following conditions
 *   are met:
 *
 *     * Redistributions of source code must retain the above copyright
 *       notice, this list of conditions and the following disclaimer.
 *     * Redistributions in binary form must reproduce the above copyright
 *       notice, this list of conditions and the following disclaimer in
 *       the documentation and/or other materials provided with the
 *       distribution.
 *     * Neither the name of Intel Corporation nor the names of its
 *       contributors may be used to endorse or promote products derived
 *       from this software without specific prior written permission.
 *
 *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

#ifndef _VIRTIO_RING_H_
#define _VIRTIO_RING_H_

#include <stdint.h>

// #include <rte_common.h>
#define RTE_PTR_ADD(ptr, x) ((void*)((uintptr_t)(ptr) + (x)))
#define RTE_ALIGN_FLOOR(val, align) (typeof(val))((val) & (~((typeof(val))((align)-1))))
#define RTE_ALIGN_CEIL(val, align) RTE_ALIGN_FLOOR(((val) + ((typeof(val))(align)-1)), align)
#define RTE_PTR_ALIGN_FLOOR(ptr, align) ((typeof(ptr))RTE_ALIGN_FLOOR((uintptr_t)ptr, align))
#define RTE_PTR_ALIGN_CEIL(ptr, align) RTE_PTR_ALIGN_FLOOR((typeof(ptr))RTE_PTR_ADD(ptr, (align)-1), align)

/*
 * VirtIO Header, located in BAR 0.
 */
#define VIRTIO_PCI_HOST_FEATURES 0  /* host's supported features (32bit, RO)*/
#define VIRTIO_PCI_GUEST_FEATURES 4 /* guest's supported features (32, RW) */
#define VIRTIO_PCI_QUEUE_PFN 8      /* physical address of VQ (32, RW) */
#define VIRTIO_PCI_QUEUE_NUM 12     /* number of ring entries (16, RO) */
#define VIRTIO_PCI_QUEUE_SEL 14     /* current VQ selection (16, RW) */
#define VIRTIO_PCI_QUEUE_NOTIFY 16  /* notify host regarding VQ (16, RW) */
#define VIRTIO_PCI_STATUS 18        /* device status register (8, RW) */
#define VIRTIO_PCI_ISR 19           /* interrupt status register, reading also clears the register (8, RO) */
/* Only if MSIX is enabled: */
#define VIRTIO_MSI_CONFIG_VECTOR 20 /* configuration change vector (16, RW) */
#define VIRTIO_MSI_QUEUE_VECTOR 22  /* vector for selected VQ notifications (16, RW) */

/* Status byte for guest to report progress. */
#define VIRTIO_CONFIG_STATUS_RESET 0x00
#define VIRTIO_CONFIG_STATUS_ACK 0x01
#define VIRTIO_CONFIG_STATUS_DRIVER 0x02
#define VIRTIO_CONFIG_STATUS_DRIVER_OK 0x04
#define VIRTIO_CONFIG_STATUS_FEATURES_OK 0x08
#define VIRTIO_CONFIG_STATUS_FAILED 0x80

/*
 * How many bits to shift physical queue address written to QUEUE_PFN.
 * 12 is historical, and due to x86 page size.
 */
#define VIRTIO_PCI_QUEUE_ADDR_SHIFT 12

/* This marks a buffer as continuing via the next field. */
#define VRING_DESC_F_NEXT 1
/* This marks a buffer as write-only (otherwise read-only). */
#define VRING_DESC_F_WRITE 2
/* This means the buffer contains a list of buffer descriptors. */
#define VRING_DESC_F_INDIRECT 4

/* The feature bitmap for virtio net */
#define VIRTIO_NET_F_CSUM 0            /* Host handles pkts w/ partial csum */
#define VIRTIO_NET_F_GUEST_CSUM 1      /* Guest handles pkts w/ partial csum */
#define VIRTIO_NET_F_MTU 3             /* Initial MTU advice. */
#define VIRTIO_NET_F_MAC 5             /* Host has given MAC address. */
#define VIRTIO_NET_F_GUEST_TSO4 7      /* Guest can handle TSOv4 in. */
#define VIRTIO_NET_F_GUEST_TSO6 8      /* Guest can handle TSOv6 in. */
#define VIRTIO_NET_F_GUEST_ECN 9       /* Guest can handle TSO[6] w/ ECN in. */
#define VIRTIO_NET_F_GUEST_UFO 10      /* Guest can handle UFO in. */
#define VIRTIO_NET_F_HOST_TSO4 11      /* Host can handle TSOv4 in. */
#define VIRTIO_NET_F_HOST_TSO6 12      /* Host can handle TSOv6 in. */
#define VIRTIO_NET_F_HOST_ECN 13       /* Host can handle TSO[6] w/ ECN in. */
#define VIRTIO_NET_F_HOST_UFO 14       /* Host can handle UFO in. */
#define VIRTIO_NET_F_MRG_RXBUF 15      /* Host can merge receive buffers. */
#define VIRTIO_NET_F_STATUS 16         /* virtio_net_config.status available */
#define VIRTIO_NET_F_CTRL_VQ 17        /* Control channel available */
#define VIRTIO_NET_F_CTRL_RX 18        /* Control channel RX mode support */
#define VIRTIO_NET_F_CTRL_VLAN 19      /* Control channel VLAN filtering */
#define VIRTIO_NET_F_CTRL_RX_EXTRA 20  /* Extra RX mode control support */
#define VIRTIO_NET_F_GUEST_ANNOUNCE 21 /* Guest can announce device on the network */
#define VIRTIO_NET_F_MQ 22             /* Device supports Receive Flow Steering */
#define VIRTIO_NET_F_CTRL_MAC_ADDR 23  /* Set MAC address */

/* Do we get callbacks when the ring is completely used, even if we've suppressed them? */
#define VIRTIO_F_NOTIFY_ON_EMPTY 24

/* Can the device handle any descriptor layout? */
#define VIRTIO_F_ANY_LAYOUT 27

/* We support indirect buffer descriptors */
#define VIRTIO_RING_F_INDIRECT_DESC 28

#define VIRTIO_F_VERSION_1 32
#define VIRTIO_F_IOMMU_PLATFORM 33

/**
 * Control the RX mode, ie. promiscuous, allmulti, etc...
 * All commands require an "out" sg entry containing a 1 byte
 * state value, zero = disable, non-zero = enable.  Commands
 * 0 and 1 are supported with the VIRTIO_NET_F_CTRL_RX feature.
 * Commands 2-5 are added with VIRTIO_NET_F_CTRL_RX_EXTRA.
 */
#define VIRTIO_NET_CTRL_RX 0
#define VIRTIO_NET_CTRL_RX_PROMISC 0
#define VIRTIO_NET_CTRL_RX_ALLMULTI 1
#define VIRTIO_NET_CTRL_RX_ALLUNI 2
#define VIRTIO_NET_CTRL_RX_NOMULTI 3
#define VIRTIO_NET_CTRL_RX_NOUNI 4
#define VIRTIO_NET_CTRL_RX_NOBCAST 5

struct virtio_net_ctrl_hdr {
	uint8_t class;
	uint8_t cmd;
} __attribute__((packed));

typedef uint8_t virtio_net_ctrl_ack;

#define VIRTIO_NET_OK 0
#define VIRTIO_NET_ERR 1

#define VIRTIO_MAX_CTRL_DATA 2048

struct virtio_pmd_ctrl {
	struct virtio_net_ctrl_hdr hdr;
	virtio_net_ctrl_ack status;
	uint8_t data[VIRTIO_MAX_CTRL_DATA];
};

/**
 * This is the first element of the scatter-gather list.  If you don't
 * specify GSO or CSUM features, you can simply ignore the header.
 */
struct virtio_legacy_net_hdr {
#define VIRTIO_NET_HDR_F_NEEDS_CSUM 1 /**< Use csum_start,csum_offset*/
#define VIRTIO_NET_HDR_F_DATA_VALID 2 /**< Checksum is valid */
	uint8_t flags;
#define VIRTIO_NET_HDR_GSO_NONE 0   /**< Not a GSO frame */
#define VIRTIO_NET_HDR_GSO_TCPV4 1  /**< GSO frame, IPv4 TCP (TSO) */
#define VIRTIO_NET_HDR_GSO_UDP 3    /**< GSO frame, IPv4 UDP (UFO) */
#define VIRTIO_NET_HDR_GSO_TCPV6 4  /**< GSO frame, IPv6 TCP */
#define VIRTIO_NET_HDR_GSO_ECN 0x80 /**< TCP has ECN set */
	uint8_t gso_type;
	uint16_t hdr_len;     /**< Ethernet + IP + tcp/udp hdrs */
	uint16_t gso_size;    /**< Bytes to append to hdr_len per frame */
	uint16_t csum_start;  /**< Position to start checksumming from */
	uint16_t csum_offset; /**< Offset after that to place checksum */
};

/* This marks a buffer as continuing via the next field. */
#define VRING_DESC_F_NEXT 1
/* This marks a buffer as write-only (otherwise read-only). */
#define VRING_DESC_F_WRITE 2
/* This means the buffer contains a list of buffer descriptors. */
#define VRING_DESC_F_INDIRECT 4

/* The Host uses this in used->flags to advise the Guest: don't kick me
 * when you add a buffer.  It's unreliable, so it's simply an
 * optimization.  Guest will still kick if it's out of buffers. */
#define VRING_USED_F_NO_NOTIFY 1
/* The Guest uses this in avail->flags to advise the Host: don't
 * interrupt me when you consume a buffer.  It's unreliable, so it's
 * simply an optimization. */
#define VRING_AVAIL_F_NO_INTERRUPT 1

/* VirtIO ring descriptors: 16 bytes.
 * These can chain together via "next". */
struct vring_desc {
	uint64_t addr;  /*  Address (guest-physical). */
	uint32_t len;   /* Length. */
	uint16_t flags; /* The flags as indicated above. */
	uint16_t next;  /* We chain unused descriptors via this. */
};

struct vring_avail {
	uint16_t flags;
	uint16_t idx;
	uint16_t ring[0];
};

/* id is a 16bit index. uint32_t is used here for ids for padding reasons. */
struct vring_used_elem {
	/* Index of start of used descriptor chain. */
	uint32_t id;
	/* Total length of the descriptor chain which was written to. */
	uint32_t len;
};

struct vring_used {
	uint16_t flags;
	volatile uint16_t idx;
	struct vring_used_elem ring[0];
};

struct vring {
	unsigned int num;
	struct vring_desc* desc;
	struct vring_avail* avail;
	struct vring_used* used;
};

/* The standard layout for the ring is a continuous chunk of memory which
 * looks like this.  We assume num is a power of 2.
 *
 * struct vring {
 *      // The actual descriptors (16 bytes each)
 *      struct vring_desc desc[num];
 *
 *      // A ring of available descriptor heads with free-running index.
 *      uint16_t avail_flags;
 *      uint16_t avail_idx;
 *      uint16_t available[num];
 *      uint16_t used_event_idx;
 *
 *      // Padding to the next align boundary.
 *      char pad[];
 *
 *      // A ring of used descriptor heads with free-running index.
 *      uint16_t used_flags;
 *      uint16_t used_idx;
 *      struct vring_used_elem used[num];
 *      uint16_t avail_event_idx;
 * };
 *
 * NOTE: for VirtIO PCI, align is 4096.
 */

struct virtqueue {
	struct vring vring; // The DMA'd region containing the descriptors etc.
	// Additional information for the driver only
	uint64_t notification_offset;
	uint16_t vq_used_last_idx;
	struct mempool* mempool; // Unused in Tx queues
	// virtual addresses to map descriptors back to their mbuf for freeing
	void* virtual_addresses[];
};

/*
 * We publish the used event index at the end of the available ring, and vice
 * versa. They are at the end for backwards compatibility.
 */
#define vring_used_event(vr) ((vr)->avail->ring[(vr)->num])
#define vring_avail_event(vr) (*(uint16_t*)&(vr)->used->ring[(vr)->num])

/*
 * The following is used with VIRTIO_RING_F_EVENT_IDX.
 * Assuming a given event_idx value from the other size, if we have
 * just incremented index from old to new_idx, should we trigger an
 * event?
 */
static inline int vring_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old) {
	return (uint16_t)(new_idx - event_idx - 1) < (uint16_t)(new_idx - old);
}

#endif /* _VIRTIO_RING_H_ */