🚀 Kernel bypass & the user-space stack
Onload's core idea: host TCP in user space with direct NIC queues, not a TCP-offload engine — and the machinery that keeps it correct.
2.1 The Onload model
The shape of the idea. A normal TCP send crosses into the kernel, which copies your bytes into a socket buffer, runs the TCP/IP state machine in kernel context, builds the packet, and hands it to the driver. Every step costs something: a syscall trap, at least one copy, an interrupt on completion, and a steady eviction of your application's hot data out of L1/L2 as the kernel runs its own code and touches its own structures. For a 64-byte message that whole apparatus is pure overhead. The Onload model deletes most of it by doing the TCP/IP work in the application's own address space and on the application's own core.
`US8489761B2` — Onload network protocol stacks. This is the cleanest patent-level statement of that idea. Protocol processing is driven from a user-mode socket/transport library: the library forms the TCP/IP packet in application context and places it directly in memory the NIC can read, then triggers the NIC to send. No per-packet trip through the kernel stack. The patent is explicit that this is not a full TCP-offload engine — the protocol logic stays in host software (the user-level library), so you keep TCP's flexibility and correctness rather than freezing a state machine into NIC silicon. It also names the costs it is attacking directly: context switches, copies, interrupts, and cache pollution, especially for small packets where those fixed costs dominate. That framing is the whole pitch in one sentence: keep the sockets programming model, move the hot path out of the kernel, and let the NIC be a fast queue rather than a protocol processor.
`US8612536B2` — the user-level stack, and the safe-fallback half. Doing TCP in user space raises an awkward question: what happens when the application that owns the connection is descheduled, blocked, or simply crashes? TCP does not stop having obligations just because your thread isn't running — ACKs are due, retransmit timers fire, the peer expects progress. US8612536B2 describes the user-level transport library working against buffers that the operating system can also access and recover. NIC timer and failure events can pull the kernel back into the loop when the application cannot make progress on its own, so the connection stays correct even though the common-case datapath never touches the kernel. This is the structural reason kernel bypass is shippable rather than a benchmark trick: the OS remains the authority that can step in, reclaim shared buffers, and keep the protocol honest when the fast path stalls.
Why this matters for low-level NIC work. The cost model here is the mental model you carry into every datapath decision. You are not chasing "the kernel is slow"; you are counting concrete events per packet — syscalls, copies, interrupts, cache lines touched — and trying to drive each toward zero on the hot path while leaving a trusted owner for the rare-but-mandatory cases (failure, descheduling, resource reclamation). That partition — latency-critical work in user space, correctness and recovery in trusted software — is the recurring discipline across the entire portfolio. It is also exactly the boundary an interviewer will probe: which invariants must the kernel or NIC keep even when the data path is in user space, and where does fallback live.
Sources
2.2 Staying correct & transparent
The transparency contract. Onload's selling point is that you do not rewrite your application. You LD_PRELOAD a library, and existing code that calls socket(), connect(), send(), recv(), and epoll_wait() simply runs faster. That only works if the acceleration layer reproduces POSIX semantics exactly — including the awkward edges that synthetic packet benchmarks never exercise: routing changes mid-connection, ARP/neighbour updates, file-descriptor inheritance across fork/exec, blocking-vs-nonblocking behaviour, and event-loop ordering. The two patents below are about making real applications work, not about moving bytes faster.
`US8533740B2` — intercepting instructions, and a shadow of the kernel's routing state. A user-level TCP stack still needs facts that the kernel owns. To send a packet it must answer: which interface, which next-hop MAC, which source address? In the kernel those answers come from the routing and neighbour tables. The naive approach — ask the kernel on every send — reintroduces the syscall you just worked so hard to avoid. US8533740B2 covers the support machinery around the user-level stack: it intercepts the relevant instructions/syscalls and maintains a copy of the OS network routing state for the accelerated path to consult. The fast path makes its forwarding decision against this shadow without a kernel lookup per packet, while the interception layer keeps the shadow consistent with the real kernel state as it changes. This is the concrete mechanism behind "transparent" acceleration: the application keeps using ordinary socket calls, the stack shadows just enough kernel-derived truth to decide correctly, and a route change does not silently break a live connection or send packets out the wrong interface.
`US9384071B2` — epoll optimisations, and why event loops are first-class. Modern servers are not built on one socket — they are built on an event loop polling thousands of file descriptors through epoll. If acceleration only fixed send/recv, every scalable server would still bottleneck on epoll_ctl and epoll_wait syscalls. US9384071B2 optimises this notification path for user-level stacks. It intercepts and batches many epoll_ctl configuration changes, storing the parameters at user level so they collapse into far fewer kernel crossings, and it uses a controlled busy-wait before blocking — spin briefly to catch imminent readiness with no context-switch latency, then fall back to a real block so you don't burn a core when idle. The patent names OpenOnload directly as a preferred context. The lesson is that preserving the programming model (a standard scalable event loop) is as much a part of the design as the datapath itself: the acceleration has to integrate with epoll, not replace it.
Why this matters for low-level NIC work. These two patents are the difference between a demo and a product. A user-space stack that ignores routing changes or breaks epoll semantics will pass a latency microbenchmark and then corrupt a real deployment in subtle, intermittent ways — exactly the bugs that are hardest to make "boring." For an engineer ramping into datapath work, the takeaway is that the hard part of kernel bypass is not the fast path; it is faithfully shadowing the kernel's view of the world (routing, neighbours, descriptor lifecycle, event readiness) so the fast path can stay fast and correct. Correctness and POSIX fidelity are the constraint that the whole architecture is built around.
Sources
2.3 Why locality, not just syscalls
Once the syscalls are gone, the bottleneck moves. It is tempting to think kernel bypass is finished once the trap, the copy, and the interrupt are eliminated. They are not the end of the latency budget — they are the part that was easy to see. Strip them away and the dominant costs become microarchitectural: which core touches which socket's state, how often a cache line bounces between cores, and how much time threads spend contending for a lock guarding shared structures. On a modern multi-core machine, a cache line ping-ponging between two cores' L1 caches, or a contended lock serialising work that should be parallel, can cost more than the syscall you just removed. Fast networking, at this point, is a CPU/cache-locality problem as much as a syscall problem.
`US9304825B2` — per-core socket state. This patent attacks exactly that. Scaling receive processing across cores with RSS is only a win if the per-flow work on each core stays local to that core. The classic failure mode: one socket's state — its accept queue, its SYN-received queue, its receive queue, its transmit state — is shared, so every core that touches the socket must acquire the lock and pull the cache lines to itself, and the lines bounce continuously between cores. The parallelism is nominal; the contention is real, and tail latency spikes. US9304825B2 describes maintaining per-core or per-core-group instances of that socket state. A core works against its own copy of the accept/SYN/receive/transmit structures, so the common case needs no cross-core lock and no shared cache line. Contention and cache-line bouncing drop out of the hot path, and the cores actually run in parallel instead of taking turns.
Why this is the natural next layer. This is the same partition philosophy as the Onload core, pushed one level deeper. The Onload patents move the datapath out of the kernel; this one keeps the datapath from tripping over itself across cores. Both are about removing a shared bottleneck from the common case while leaving a correct path for the rare case where state genuinely must be combined. And it ties the whole chapter together: the win from kernel bypass is not automatic. You can delete every syscall and still lose all of it to a lock that serialises your cores or a hot cache line that bounces on every packet.
Why this matters for low-level NIC work. This is the layer where an interviewer separates people who have only read about kernel bypass from people who have profiled it. The practical questions are concrete: pin the application thread to the core whose RSS queue the NIC steers its flow to, so the packet, the descriptor ring, the socket state, and the consuming code all live in one core's cache and one NUMA node. When tail latency regresses, the diagnosis is rarely "syscalls" — it is steering sending a flow to the wrong core, a lock under contention, a cache line shared across a NUMA boundary, or the application thread scheduled away from its data. The mental model to carry in: affinity, cache locality, and lock contention are first-class networking concerns, co-equal with avoiding the kernel — and the patent record shows the team treated them that way from early on.
Sources