← All files328 lineson GitHub ↗

src/libixy-vfio.c

VFIO / IOMMU: groups, containers, and IOVA DMA mappings.

Walkthrough, interview notes & deep dive

VFIO (Virtual Function I/O) is the modern standard for building high-performance userspace drivers in Linux, succeeding older and less secure frameworks like UIO. For an AMD or Solarflare engineer, understanding this file is critical because it manages the boundary between the CPU and the NIC hardware. This implementation handles the heavy lifting of IOMMU group management, DMA mapping, and device register access, ensuring that a userspace application can control a PCIe device without compromising the stability or security of the rest of the system.

The core responsibility of this file is to bridge the gap between userspace memory and hardware DMA. In a high-frequency trading or low-latency networking context, you cannot afford the overhead of kernel-mediated packet processing. However, giving a userspace process raw access to hardware is dangerous. VFIO solves this by leveraging the IOMMU (Input-Output Memory Management Unit). The code here orchestrates a multi-step handshake: identifying the IOMMU group of a device, attaching it to a "container" (which represents the IOMMU context), and then programming the IOMMU to allow the device to "see" specific slices of process memory.

int cfd = open("/dev/vfio/vfio", O_RDWR);
ioctl(cfd, VFIO_GET_API_VERSION);
ioctl(cfd, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU);
// ...
int vfio_gfd = check_err(open(path, O_RDWR), "open vfio group");
check_err(ioctl(vfio_gfd, VFIO_GROUP_SET_CONTAINER, &cfd), "set container");
ret = check_err(ioctl(cfd, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU), "set IOMMU type");
int vfio_fd = check_err(ioctl(vfio_gfd, VFIO_GROUP_GET_DEVICE_FD, pci_addr), "get device fd");

The initialization sequence in vfio_init follows the strict Linux VFIO API. It first determines the IOMMU group ID by reading the symbolic links in /sys/bus/pci/devices/. It then opens the VFIO container (/dev/vfio/vfio) and the specific group file (e.g., /dev/vfio/12). By calling VFIO_GROUP_SET_CONTAINER, the code binds the group to the container's translation context. This is a security boundary: the IOMMU ensures that no device in that group can access memory outside of what is explicitly mapped in that container.

Once the device file descriptor is obtained via VFIO_GROUP_GET_DEVICE_FD, the driver can interact with the NIC's registers. The function vfio_map_region retrieves information about the device's BARs (Base Address Registers) using VFIO_DEVICE_GET_REGION_INFO. It then maps these hardware registers directly into the process's virtual address space using mmap on the device file descriptor. This allows the driver to perform MMIO (Memory-Mapped I/O) by simply reading and writing to a memory pointer, bypassing the kernel for every register access.

VFIO/IOMMU: group to container to device, then IOVA DMA mappings
VFIO/IOMMU: group to container to device, then IOVA DMA mappings

The safety of userspace DMA is the primary reason to use VFIO over UIO. In vfio_map_dma, the code uses the VFIO_IOMMU_MAP_DMA ioctl to register a memory region for hardware access. When a NIC performs DMA, it doesn't use the CPU's MMU; it issues requests to the PCIe bus. Without an IOMMU, the NIC would use raw physical addresses. A buggy userspace driver could accidentally point the NIC at the kernel's memory, causing a catastrophic crash or data leak.

The IOMMU acts as a "device-side TLB." When vfio_map_dma is called, the kernel pins the requested pages (preventing them from being swapped) and programs the IOMMU with a mapping between an IOVA (I/O Virtual Address) and the actual physical address. If the NIC attempts to access an address that hasn't been explicitly mapped via VFIO_IOMMU_MAP_DMA, the IOMMU blocks the transaction and triggers a fault, effectively confining the device to a "sandbox" of memory.

struct vfio_iommu_type1_dma_map dma_map = {
    .vaddr = (uint64_t) vaddr,
    .iova = iova, // In this driver, IOVA is usually set to vaddr
    .size = size < MIN_DMA_MEMORY ? MIN_DMA_MEMORY : size,
    .argsz = sizeof(dma_map),
    .flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE};
int cfd = get_vfio_container();
check_err(ioctl(cfd, VFIO_IOMMU_MAP_DMA, &dma_map), "IOMMU Map DMA Memory");

IOVA vs. Physical Address is a distinction that often comes up in low-level interviews. A Physical Address (PA) is the actual location in the RAM chips. An IOVA is a virtual address used by the device on the PCIe bus. In this implementation, the iova is typically set to the process's virtual address (vaddr) for simplicity. This creates a 1:1 mapping between the pointer the C code uses and the address the NIC uses. However, the IOMMU translates this IOVA into the real PA behind the scenes. This abstraction is what allows userspace drivers to work safely without knowing the underlying physical layout of the machine.

Finally, the file handles interrupt delivery via vfio_enable_msix and eventfd. Traditional hardware interrupts cannot be delivered directly to userspace. Instead, VFIO uses an eventfd to signal the application. The driver creates an eventfd, registers it with VFIO using VFIO_DEVICE_SET_IRQS, and then uses epoll or a blocking read to wait for hardware events like packet arrival. This completes the loop: the driver can send packets via MMIO, the device can access data via IOMMU-protected DMA, and the application is notified of completion via eventfd-backed interrupts.

Interview angles

  • What is the primary role of the IOMMU in VFIO? The IOMMU provides memory isolation and protection. It translates I/O Virtual Addresses (IOVAs) to Physical Addresses (PAs) and ensures the hardware device cannot access memory regions that haven't been explicitly mapped by the driver, preventing memory corruption from buggy or malicious userspace code.
  • How does VFIO differ from UIO? UIO is simpler but less secure; it requires the driver to handle raw physical addresses and offers no protection against DMA-based attacks or bugs. VFIO uses the IOMMU for protection, supports DMA mapping of arbitrary userspace memory, and provides a unified interface for interrupts and BAR access.
  • What is an IOVA and why do we use it? An I/O Virtual Address is the address a device uses to access memory over the PCIe bus. We use it to decouple the device's view of memory from the CPU's view. By mapping the process's virtual address as the IOVA, we allow the driver to pass its own pointers directly to the hardware, while the IOMMU handles the translation to physical RAM.

Going deeper

uint16_t dma = 0;
assert(pread(device_fd, &dma, 2, conf_reg.offset + command_register_offset) == 2);
dma |= 1 << bus_master_enable_bit;
assert(pwrite(device_fd, &dma, 2, conf_reg.offset + command_register_offset) == 2);

PCIe Command Register modification. While BARs (Base Address Registers) are usually mmap'd for performance, the PCIe Configuration Space is accessed via pread/pwrite on the device_fd. The code targets offset 4 (Command Register) and flips bit 2 (Bus Master Enable). Without this, the NIC cannot initiate DMA transactions toward host memory, regardless of IOMMU settings. The 2-byte size is critical; Configuration Space registers have specific access widths (1, 2, or 4 bytes), and an incorrect width can trigger a Target Abort.

#define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + sizeof(int) * (MAX_INTERRUPT_VECTORS + 1))
...
char irq_set_buf[MSIX_IRQ_SET_BUF_LEN];
irq_set = (struct vfio_irq_set*) irq_set_buf;
fd_ptr = (int*) &irq_set->data;
fd_ptr[0] = event_fd;

Flexible array member over-allocation. The vfio_irq_set struct ends with a data[] flexible array member. This code uses a stack-allocated char buffer to manually provide contiguous space for the int file descriptors (the eventfd handles). By casting the char array to the struct pointer, the driver can treat the trailing memory as an array of FDs. This pattern allows a single ioctl to configure multiple interrupt vectors simultaneously, minimizing syscall overhead during initialization.

for (int i = 0; i < rc; i++) {
    uint64_t val;
    check_err(read(events[i].data.fd, &val, sizeof(val)), "to read event");
}

Eventfd drain requirement. When a VFIO interrupt triggers, the kernel increments an internal 64-bit counter associated with the eventfd. Because the epoll instance is likely level-triggered or requires re-arming, the driver must perform an 8-byte read() to reset the counter to zero. If this read is skipped, epoll_wait will immediately return again in a "busy-loop" because the file descriptor remains in a readable state, even if no new interrupt has occurred.

Harder interview questions

  • Why does `vfio_init` check `VFIO_GROUP_FLAGS_VIABLE`? VFIO enforces isolation at the IOMMU group level, not the individual device level. If multiple devices (e.g., a multi-port NIC or a GPU and its audio controller) share a group, all must be bound to VFIO drivers. If even one device is still controlled by a kernel driver, the group is not "viable," and the container will refuse to attach to prevent cross-device memory leakage.
  • What is the significance of the `argsz` field in every VFIO struct? This is a UAPI versioning mechanism. The user sets argsz to the size of the struct they are passing. The kernel checks this to ensure compatibility; if the kernel supports a newer, larger version of the struct, it uses argsz to avoid over-reading. If the kernel returns a value larger than the provided argsz, it signals that the buffer was too small to receive all available capability data.
  • Why is `MIN_DMA_MEMORY` set to 4096 bytes? The IOMMU operates on page granularity (typically 4KB). VFIO_IOMMU_MAP_DMA requires the size and iova (IO Virtual Address) to be page-aligned. Attempting to map a smaller buffer (like a 64-byte descriptor ring) would fail or force the kernel to map the entire surrounding page, potentially exposing adjacent private data to the device's DMA engine.
  • Explain the `firstsetup` logic in `vfio_init`. A single VFIO container can manage multiple device groups. The IOMMU type (VFIO_TYPE1_IOMMU) must be set exactly once for the container, and only after at least one group has been attached. The get_vfio_container helper allows the process to share one container across multiple NICs, ensuring they all share the same IOMMU address space and translation tables.

Gotchas

  • The off-by-one in MSI-X clamping. The code clamps interrupt_vector to MAX_INTERRUPT_VECTORS + 1. If you request exactly 32 vectors, but the hardware only supports 16, the ioctl will fail. You must use VFIO_DEVICE_GET_IRQ_INFO first to determine the hardware's actual capacity.
  • Config space vs. MMIO. Never attempt to mmap the PCIe config space (Region 7). Always use pread/pwrite. Only BAR regions (0-5) are safely mmap-able.
  • Eventfd ownership. The event_fd created in vfio_enable_msix is passed to the kernel, but the userspace process is responsible for closing it. If the driver is reloaded without closing these FDs, you will leak file descriptors and eventually hit RLIMIT_NOFILE.

From ixy to a production driver

libixy-vfio.c exposes work production stacks usually hide: it opens the VFIO container and group, checks group viability, selects VFIO_TYPE1_IOMMU, mmaps BARs, sets the PCIe Command register Bus Master Enable bit, installs VFIO_IOMMU_MAP_DMA entries, and turns MSI/MSI-X interrupts into eventfds watched by epoll. In ixgbe.ko, the same jobs are split across the driver core, PCI subsystem, DMA API, IRQ subsystem, and netdev/NAPI layers. The interview point is that ixy is deliberately narrow so the mechanism is visible.

  • DMA setup: ixy maps each packet buffer with IOVA equal to the process virtual address, a 1:1 userspace VA scheme. That makes descriptor programming simple, but pinned pages and IOMMU updates are expensive. A kernel driver allocates descriptor rings with dma_alloc_coherent() and maps packet buffers with dma_map_single() or related streaming APIs, including direction flags such as DMA_TO_DEVICE and DMA_FROM_DEVICE, cache synchronization, and dma_unmap_*() lifetime rules. Kernel IOMMU behavior is policy controlled: iommu.strict=1 invalidates IOMMU TLBs synchronously on unmap, iommu.strict=0 permits lazy/deferred invalidation, and iommu.passthrough=1 can bypass translation by default. DPDK is closer to ixy because it also uses vfio-pci, but it chooses IOVA-as-VA or IOVA-as-PA at EAL startup and amortizes mapping cost with hugepages and large memseg mappings instead of pinning tiny buffers individually.
  • Device ownership: ixy performs VFIO container/group plumbing itself because userspace must prove the IOMMU group is isolated before DMA is allowed. In the kernel path, binding ixgbe.ko gives the PCI core and probe path that role; enabling DMA is a pci_set_master() call, which sets the bus-master bit in PCI_COMMAND, rather than open-coded config-space mutation. With DPDK, the NIC is rebound away from ixgbe to vfio-pci, so VFIO group isolation still matters, but EAL centralizes setup for all PMDs.
  • Interrupts and queues: ixy wires one MSI/MSI-X vector to one eventfd and blocks in epoll. A production Linux NIC driver uses MSI-X per queue, RSS to spread flows, and NAPI: the interrupt handler masks or quiesces more interrupts, calls napi_schedule(), and the poll method drains a packet budget. On Intel 82599, interrupt moderation is controlled by EITR/ITR-style throttle registers; multiqueue steering uses RSS tables such as RETA. DPDK PMDs usually disable interrupts on the hot path and busy-poll Rx/Tx queues, using interrupts mostly for link status or power-saving modes.
  • What is omitted: ixy has no production concurrency model for shared MMIO, RSS/multiqueue setup, checksum offload, TSO, RSC/LRO, flow control, power management, hotplug, SR-IOV lifecycle, recoverable error paths, ethtool, or detailed stats. It tends to assert or check_err and abort. That is fine for teaching and benchmarking because it keeps the causal chain short: BAR access, DMA addressability, descriptor rings, and interrupt delivery. It is unacceptable in production because the driver must survive races, partial failures, topology changes, mixed workloads, and operational observability requirements.
VFIO: container, group, and device fds + the IOMMU DMA map.
VFIO: container, group, and device fds + the IOMMU DMA map.

Sources

Source

filesrc/libixy-vfio.c
#include <errno.h>
#include <fcntl.h>
#include <libgen.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

#include <linux/limits.h>
#include <linux/vfio.h>

#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/eventfd.h>
#include <sys/epoll.h>

#include <driver/device.h>

#define IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + sizeof(int))
#define MAX_INTERRUPT_VECTORS 32
#define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + sizeof(int) * (MAX_INTERRUPT_VECTORS + 1))

ssize_t MIN_DMA_MEMORY = 4096; // we can not allocate less than page_size memory

void vfio_enable_dma(int device_fd) {
	// write to the command register (offset 4) in the PCIe config space
	int command_register_offset = 4;
	// bit 2 is "bus master enable", see PCIe 3.0 specification section 7.5.1.1
	int bus_master_enable_bit = 2;
	// Get region info for config region
	struct vfio_region_info conf_reg = {.argsz = sizeof(conf_reg)};
	conf_reg.index = VFIO_PCI_CONFIG_REGION_INDEX;
	check_err(ioctl(device_fd, VFIO_DEVICE_GET_REGION_INFO, &conf_reg), "get vfio config region info");
	uint16_t dma = 0;
	assert(pread(device_fd, &dma, 2, conf_reg.offset + command_register_offset) == 2);
	dma |= 1 << bus_master_enable_bit;
	assert(pwrite(device_fd, &dma, 2, conf_reg.offset + command_register_offset) == 2);
}

/**
 * Enable VFIO MSI interrupts.
 * @param device_fd The VFIO file descriptor.
 * @return The event file descriptor.
 */
int vfio_enable_msi(int device_fd) {
	info("Enable MSI Interrupts");
	char irq_set_buf[IRQ_SET_BUF_LEN];
	int* fd_ptr;

	// setup event fd
	int event_fd = eventfd(0, 0);

	struct vfio_irq_set* irq_set = (struct vfio_irq_set*) irq_set_buf;
	irq_set->argsz = sizeof(irq_set_buf);
	irq_set->count = 1;
	irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
	irq_set->index = VFIO_PCI_MSI_IRQ_INDEX;
	irq_set->start = 0;
	fd_ptr = (int*) &irq_set->data;
	*fd_ptr = event_fd;

	check_err(ioctl(device_fd, VFIO_DEVICE_SET_IRQS, irq_set), "enable MSI interrupts");

	return event_fd;
}

/**
 * Disable VFIO MSI interrupts.
 * @param device_fd The VFIO file descriptor.
 * @return 0 on success.
 */
int vfio_disable_msi(int device_fd) {
	info("Disable MSI Interrupts");
	char irq_set_buf[IRQ_SET_BUF_LEN];

	struct vfio_irq_set* irq_set = (struct vfio_irq_set*) irq_set_buf;
	irq_set->argsz = sizeof(irq_set_buf);
	irq_set->count = 0;
	irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
	irq_set->index = VFIO_PCI_MSI_IRQ_INDEX;
	irq_set->start = 0;

	check_err(ioctl(device_fd, VFIO_DEVICE_SET_IRQS, irq_set), "disable MSI interrupts");

	return 0;
}

/**
 * Enable VFIO MSI-X interrupts.
 * @param device_fd The VFIO file descriptor.
 * @return The event file descriptor.
 */
int vfio_enable_msix(int device_fd, uint32_t interrupt_vector) {
	info("Enable MSIX Interrupts");
	char irq_set_buf[MSIX_IRQ_SET_BUF_LEN];
	struct vfio_irq_set* irq_set;
	int* fd_ptr;

	// setup event fd
	int event_fd = eventfd(0, 0);

	irq_set = (struct vfio_irq_set*) irq_set_buf;
	irq_set->argsz = sizeof(irq_set_buf);
	if (!interrupt_vector) {
		interrupt_vector = 1;
	} else if (interrupt_vector > MAX_INTERRUPT_VECTORS)
		interrupt_vector = MAX_INTERRUPT_VECTORS + 1;

	irq_set->count = interrupt_vector;
	irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER;
	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
	irq_set->start = 0;
	fd_ptr = (int*) &irq_set->data;
	fd_ptr[0] = event_fd;

	check_err(ioctl(device_fd, VFIO_DEVICE_SET_IRQS, irq_set), "enable MSIX interrupt");

	return event_fd;
}

/**
 * Disable VFIO MSI-X interrupts.
 * @param device_fd The VFIO file descriptor.
 * @return 0 on success.
 */
int vfio_disable_msix(int device_fd) {
	info("Disable MSIX Interrupts");
	struct vfio_irq_set* irq_set;
	char irq_set_buf[MSIX_IRQ_SET_BUF_LEN];

	irq_set = (struct vfio_irq_set*) irq_set_buf;
	irq_set->argsz = sizeof(struct vfio_irq_set);
	irq_set->count = 0;
	irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;
	irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;
	irq_set->start = 0;

	check_err(ioctl(device_fd, VFIO_DEVICE_SET_IRQS, irq_set), "disable MSIX interrupt");

	return 0;
}

/**
 * Setup VFIO interrupts by detecting which interrupts this device supports.
 * @param device_fd The VFIO file descriptor.
 * @return The supported interrupt.
 */
int vfio_setup_interrupt(int device_fd) {
	info("Setup VFIO Interrupts");

	for (int i = VFIO_PCI_MSIX_IRQ_INDEX; i >= 0; i--) {
		struct vfio_irq_info irq = {.argsz = sizeof(irq), .index = i};

		check_err(ioctl(device_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq), "get IRQ Info");

		/* if this vector cannot be used with eventfd continue with next*/
		if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0) {
			debug("IRQ doesn't support Event FD");
			continue;
		}

		return i;
	}

	return -1;
}

/**
 * Waits for events on the epoll instance referred to by the file descriptor epoll_fd.
 * The memory area pointed to by events will contain the events that will be available for the caller.
 * Up to maxevents are returned by epoll_wait.
 * @param epoll_fd The epoll file descriptor.
 * @param maxevents The maximum number of events to return. The maxevents argument must be greater than zero.
 * @param timeout The timeout argument specifies the minimum number of milliseconds that epoll_wait will block.
 * Specifying a timeout of -1 causes epoll_wait to block indefinitely,
 * while specifying a timeout equal to zero cause epoll_wait to return immediately, even if no events are available.
 * @return Number of ready file descriptors.
 */
int vfio_epoll_wait(int epoll_fd, int maxevents, int timeout) {
	struct epoll_event events[maxevents];
	int rc;

	while (1) {
		// Waiting for packets
		rc = (int) check_err(epoll_wait(epoll_fd, events, maxevents, timeout), "to handle epoll wait");
		if (rc > 0) {
			/* epoll_wait has at least one fd ready to read */
			for (int i = 0; i < rc; i++) {
				uint64_t val;
				// read event file descriptor to clear interrupt.
				check_err(read(events[i].data.fd, &val, sizeof(val)), "to read event");
			}
			break;
		} else {
			/* rc == 0, epoll_wait timed out */
			break;
		}
	}

	return rc;
}

/**
 * Add event file descriptor to epoll.
 * @param event_fd The event file descriptor to add.
 * @return The epoll file descriptor.
 */
int vfio_epoll_ctl(int event_fd) {
	struct epoll_event event;
	event.events = EPOLLIN;
	event.data.fd = event_fd;

	int epoll_fd = (int) check_err(epoll_create1(0), "to created epoll");

	check_err(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, event_fd, &event), "to initialize epoll");

	return epoll_fd;
}

// returns the devices file descriptor or -1 on error
int vfio_init(const char* pci_addr) {
	// find iommu group for the device
	// `readlink /sys/bus/pci/device/<segn:busn:devn.funcn>/iommu_group`
	char path[PATH_MAX], iommu_group_path[PATH_MAX];
	struct stat st;
	snprintf(path, sizeof(path), "/sys/bus/pci/devices/%s/", pci_addr);
	int ret = stat(path, &st);
	if (ret < 0) {
		// No such device
		return -1;
	}
	strncat(path, "iommu_group", sizeof(path) - strlen(path) - 1);

	int len = check_err(readlink(path, iommu_group_path, sizeof(iommu_group_path)), "find the iommu_group for the device");

	iommu_group_path[len] = '\0'; // append 0x00 to the string to end it
	char* group_name = basename(iommu_group_path);
	int groupid;
	check_err(sscanf(group_name, "%d", &groupid), "convert group id to int");

	int firstsetup = 0; // Need to set up the container exactly once
	int cfd = get_vfio_container();
	if (cfd == -1) {
		firstsetup = 1;
		// open vfio file to create new vfio container
		cfd = check_err(open("/dev/vfio/vfio", O_RDWR), "open /dev/vfio/vfio");
		set_vfio_container(cfd);

		// check if the container's API version is the same as the VFIO API's
		check_err((ioctl(cfd, VFIO_GET_API_VERSION) == VFIO_API_VERSION) - 1, "get a valid API version from the container");

		// check if type1 is supported
		check_err((ioctl(cfd, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU) == 1) - 1, "get Type1 IOMMU support from the IOMMU container");
	}

	// open VFIO group containing the device
	snprintf(path, sizeof(path), "/dev/vfio/%d", groupid);
	int vfio_gfd = check_err(open(path, O_RDWR), "open vfio group");

	// check if group is viable
	struct vfio_group_status group_status = {.argsz = sizeof(group_status)};
	check_err(ioctl(vfio_gfd, VFIO_GROUP_GET_STATUS, &group_status), "get VFIO group status");
	check_err(((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) > 0) - 1, "get viable VFIO group - are all devices in the group bound to the VFIO driver?");

	// Add group to container
	check_err(ioctl(vfio_gfd, VFIO_GROUP_SET_CONTAINER, &cfd), "set container");

	if (firstsetup != 0) {
		// Set vfio type (type1 is for IOMMU like VT-d or AMD-Vi) for the
		// container.
		// This can only be done after at least one group is in the container.
		ret = check_err(ioctl(cfd, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU), "set IOMMU type");
	}

	// get device file descriptor
	int vfio_fd = check_err(ioctl(vfio_gfd, VFIO_GROUP_GET_DEVICE_FD, pci_addr), "get device fd");

	// enable DMA
	vfio_enable_dma(vfio_fd);

	return vfio_fd;
}

// returns a uint8_t pointer to the MMAPED region or MAP_FAILED if failed
uint8_t* vfio_map_region(int vfio_fd, int region_index) {
	struct vfio_region_info region_info = {.argsz = sizeof(region_info)};
	region_info.index = region_index;
	int ret = ioctl(vfio_fd, VFIO_DEVICE_GET_REGION_INFO, &region_info);
	if (ret == -1) {
		// Failed to set iommu type
		return MAP_FAILED; // MAP_FAILED == ((void *) -1)
	}
	return (uint8_t*) check_err(mmap(NULL, region_info.size, PROT_READ | PROT_WRITE, MAP_SHARED, vfio_fd, region_info.offset), "mmap vfio bar0 resource");
}

// returns iova (physical address of the DMA memory from device view) on success
uint64_t vfio_map_dma(void* vaddr, uint32_t size) {
	uint64_t iova = (uint64_t) vaddr; // map iova to process virtual address
	struct vfio_iommu_type1_dma_map dma_map = {
		.vaddr = (uint64_t) vaddr,
		.iova = iova,
		.size = size < MIN_DMA_MEMORY ? MIN_DMA_MEMORY : size,
		.argsz = sizeof(dma_map),
		.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE};
	int cfd = get_vfio_container();
	check_err(ioctl(cfd, VFIO_IOMMU_MAP_DMA, &dma_map), "IOMMU Map DMA Memory");
	return iova;
}

// unmaps previously mapped DMA region. returns 0 on success
uint64_t vfio_unmap_dma(int fd, uint64_t iova, uint32_t size) {
	struct vfio_iommu_type1_dma_unmap dma_unmap = {
		.argsz = sizeof(dma_unmap),
		.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE,
		.iova = iova,
		.size = size
	};
	int cfd = get_vfio_container();
	int ret = ioctl(cfd, VFIO_IOMMU_UNMAP_DMA, &dma_unmap);
	if (ret == -1) {
		// Failed to unmap DMA region
		return -1;
	}
	return ret;
}
← Previous file
interrupts.h
Next file →
libixy-vfio.h