⚡ ef_vi & ultra-low-latency transmit
The raw layer-2 datapath beneath ef_vi: descriptor rings and events, and the transmit tricks that shave nanoseconds.
3.1 Rings, descriptors & events
Strip away the sockets API and ef_vi is, at heart, a small set of rings in host memory plus a doorbell. The app posts transmit descriptors onto a TX ring, posts receive buffers onto an RX ring, and reads back a single event queue (the evq) to learn what the NIC has finished. The packet bytes are the obvious cost. The less obvious cost — and the one these patents attack — is the control traffic: the descriptor fetches, the doorbell writes, and the completion events that all cross PCIe. On a path measured in hundreds of nanoseconds, every extra PCIe transaction and every extra event you have to poll is part of the latency budget.
Folding "queue empty" into ordinary completions. US7831749B2 looks at how the NIC tells the host that a descriptor ring has run dry. The naive design emits a separate event for the queue-empty condition — a distinct PCIe write into the event queue, and a distinct thing the host poll loop has to recognise. The patent instead folds the queue-empty signal into the ordinary completion event the NIC was already going to post. One event now carries two facts: "this descriptor completed" and "the queue is now empty." For RX that is the more interesting case: the host learns its receive ring is draining as part of a completion it was already reading, so it can replenish buffers sooner and avoid the drop that follows an unnoticed empty ring. Fewer events on the bus, fewer events to poll, earlier replenish — all from not splitting one notification into two.
Descriptor caching and write-pointer arithmetic. US7496699B2 (a Level 5 Networks-origin patent, the company that became Solarflare) is about how the NIC actually pulls descriptors over DMA and how it tracks where the host has written. The host rings a doorbell to advance a write pointer; the NIC keeps a device-side cache of descriptors and does the pointer arithmetic to know which descriptors are newly valid and how many to prefetch. Caching descriptors on the NIC means it does not issue a fresh PCIe read for every single send — it can batch and stay ahead of the wire. The mechanism matters because a per-descriptor round trip to host memory is exactly the kind of serialized PCIe latency that sits directly in the transmit critical path.
Unified buffer memory instead of fixed FIFOs. US9008113B2 ("mapped FIFO buffering") rethinks the on-NIC packet store. Rather than rigid, fixed-size FIFOs hard-partitioned per physical port or traffic class, it uses one unified buffer memory carved into virtual queues — linked logical queues in shared memory that ingress sources (host TX, port RX) write into. The win is that buffer space follows demand: a bursty queue can borrow capacity instead of overflowing while a neighbouring fixed FIFO sits half-empty. For a low-latency datapath that means less head-of-line waste and fewer needless stalls when one flow spikes — the hardware absorbs the burst rather than dropping or backpressuring early.
The throughline across all three: at ef_vi speeds the descriptor/event/doorbell machinery is not free plumbing around the "real" work. It is a large slice of the latency, so the design fights to issue fewer PCIe transactions, cache state on the device, poll fewer events, and share buffer memory elastically.
Sources
3.2 PIO and cut-through PIO
The default transmit path is DMA: the app puts packet bytes in a buffer, posts a descriptor that points at it, rings a doorbell, and the NIC fetches first the descriptor and then the payload over PCIe. That is two device-initiated reads of host memory before a single byte hits the wire. For a 64-byte order packet, the setup dwarfs the data. The patents in this section remove that setup in two escalating steps — push the bytes to the NIC instead of pointing at them (PIO), then start sending before the bytes have all arrived (CTPIO).
Programmed I/O: write the packet straight into the NIC. US10394751B2 describes TX PIO directly. Instead of leaving the packet in host memory and handing the NIC a descriptor, the CPU writes the packet bytes itself into a buffer or scratchpad on the NIC, using ordinary store instructions to an uncached, memory-mapped I/O aperture (BAR space). A doorbell or "ready" indicator then tells the NIC the frame is complete. There is no descriptor fetch and no device-initiated DMA read of the payload — the bytes were already pushed across the PCIe write path, which is lower latency than a read round trip. The patent is careful that PIO and DMA coexist: the transmit logic chooses per packet, because PIO only wins while the packet is small. Past a size threshold the CPU store cost and the cost of the app copying into MMIO outweigh the saved descriptor fetch, and DMA's "point, don't copy" wins again. So the design knob is packet size: small, latency-critical sends go PIO; bulk goes DMA.
Cut-through PIO: start before the frame is whole. Even with PIO, a store-and-forward NIC waits until the entire frame is buffered before clocking it onto the wire — the frame is serialized once on the way in and again on the way out. US11044183B2 and its continuation US11165683B2 push the idea to cut-through: the NIC begins transmitting on the wire as soon as the leading bytes are available, overlapping receive-into-NIC with transmit-onto-wire and removing one whole frame-time of store-and-forward latency.
The danger is an underrun: if the CPU (or DMA source) cannot feed the rest of the frame fast enough, the NIC is mid-transmission with nothing left to send and cannot un-send what is already on the wire. US11044183B2 handles this by monitoring the transmit path and, on underrun, deliberately invalidating the frame — corrupting or setting a known-bad FCS/CRC — so the receiver's frame check fails and the partial garbage is silently discarded rather than delivered. A dropped frame is recoverable; a truncated-but-valid-looking frame is a correctness bug.
US11165683B2 adds the tuning knob: the NIC only begins transmitting after at least N bytes are buffered, and that start-threshold can be set per queue (or per QoS policy) and adapted. The tradeoff is direct. A low threshold means it starts almost immediately — minimum latency, but a higher chance of underrun and a wasted (deliberately corrupted) frame if the source stutters. A high threshold buffers more before committing — more latency, fewer underruns. There is no universal answer: a steady, well-fed feed handler can run an aggressive threshold; a jittery source needs a conservative one. That per-queue threshold is the latency-vs-reliability dial of the cut-through transmit path.
Sources
3.3 Templates & deadlines
The fastest send is the one whose work you already did. In a trading feed handler the moment that matters is the instant you decide to fire an order — and at that instant almost everything about the packet is already known. The Ethernet header, the IP and TCP/UDP headers, the session fields, even most of the application message are identical to the last thousand packets. Only a few late-binding fields — a price, a quantity, a sequence number — depend on the trigger. The patents here exploit that asymmetry in two ways: pre-build everything you can (templates), and refuse to send anything that has gone stale (deadlines).
Message templates: build early, patch late. US9258390B2 describes constructing a transmit message incrementally from a template. Known headers and fixed fields are assembled ahead of time, while the NIC and the user-level stack hold the partially-formed packet ready. When the late-arriving fields finally appear, they are patched into the reserved slots and the packet goes out — instead of building the whole frame, computing checksums, and walking protocol state from scratch on the critical path. The patent explicitly talks about user-level protocol stacks, NIC-side packet-processing engines, and TCP/IP over Ethernet, and notes that the small late part can be delivered by programmed I/O while a larger pre-formed part rides DMA. The continuation US10021223B2 covers the same lifecycle — create a template, update it with application data, complete the protocol headers, then have the NIC transmit — which is the natural fit for an application that repeatedly sends near-identical messages and cares about tail latency.
The latency mechanism is that you move work off the hot path and before the decision. Checksum bases, header layout, and connection state are computed during the quiet time between sends; the trigger only has to drop in a handful of bytes and ring a doorbell. The closer the template is to "done," the shorter the path from decision to wire.
Deadlines: drop stale data rather than send it late. Templates make sends faster; US9391840B2 ("avoiding delayed data") accepts that sometimes the right move is not to send at all. It associates a timestamp — effectively a freshness deadline — with data submitted for transmission, and checks near the point of transmit whether the data has exceeded a maximum age. If it has, the path can discard the packet, notify the application, or skip retransmission, according to policy, rather than putting stale bytes on the wire.
This inverts the usual reliability assumption. Normally a network stack works hard to deliver everything. But for a market order or a control signal, a late message can be worse than no message: it can execute against a price that no longer exists, or act on a world that has moved on. The patent puts explicit machinery in the datapath so that freshness can beat delivery — the system would rather drop a packet it knows is too old than serialize it onto the wire after its useful window has closed. The check sits right before transmit precisely because that is the last moment the system can still measure true latency and bail out.
Together the two ideas frame ultra-low-latency transmit as a race against a clock: do as much as possible before the gun (templates), and after the gun, send only if you are still in time (deadlines).
Sources