all postsblog / topic / internals

blog · topic

Platform internals

Millisecond sandboxes come from a stack of kernel mechanisms working together: KVM, userfaultfd, copy-on-write storage, network namespaces and cgroups. These posts go underneath the API into memory, networking, storage and scheduling, with the measurements behind the design choices.

174 posts

10 min read

KVM Memslots Explained: How a microVM's RAM Is Actually Mapped

A guest's "physical" RAM is a range of ordinary host virtual memory plus one lookup table. That table, the memslot list, is why snapshot restore, copy-on-write and memory streaming are cheap.

kvmfirecrackermemoryAjay Kumar
10 min read

virtio-net Offloads and Why Your microVM's Network Is Slower Than the Wire

Bulk transfers look fine. A sixty-four-byte request/response benchmark gets a third of bare metal. Both numbers are correct, and the gap between them is a virtio feature negotiation, a TAP device with its own opinion, and a per-packet tax that every sandbox on the host pays at once.

firecrackervirtio-netnetworkingAjay Kumar
11 min read

Flamegraphs Inside a microVM: When perf Has No Counters to Count

The flamegraph was 1,400 pixels wide and ninety-six percent of it was one box labelled [unknown]. perf runs fine inside a microVM; the recipe everybody memorised depends on three things a guest quietly does not provide, and perf will not tell you which one bit you.

perfflamegraphsprofilingAjay Kumar
11 min read

Encryption at Rest for Sandbox Disks and Snapshots

"Is customer data encrypted at rest?" is one checkbox covering four very different files, three of which are not what the reviewer is picturing. One of them is a verbatim copy of someone's RAM.

encryption-at-restsecuritysnapshotsAjay Kumar
11 min read

nftables vs iptables for Per-Sandbox Networking: Why I Still Write iptables

PandaStack writes iptables into 16,384 pre-allocated network namespaces, which is the unfashionable choice. The reason is that nftables' biggest win — not walking a long linear chain — is an argument about rule count, and our rule count per namespace is four. What actually hurts at fleet scale is the state nobody namespaced properly.

nftablesiptablesnetfilterAjay Kumar
11 min read

memory.high vs memory.max: Throttle or Kill

Hard limit and soft limit is the wrong mental model. One of these files converts an overcommit mistake into a dead process; the other converts it into a latency problem. For a multi-tenant fleet, that difference is the entire design.

cgroupslinuxmemoryAjay Kumar
11 min read

Guest Writeback Throttling: Why dirty_ratio Is Wrong for a 2 GiB MicroVM

The guest is extremely confident about a disk that does not exist. The host is also confident, about different things. Between those two confidences live your tail latencies.

firecrackerlinux-kernelwritebackAjay Kumar
11 min read

DAMON: Measuring What a Sandbox Actually Touches

Charging a host for every guest's configured RAM is what caps your density, and it is wrong by a lot — a guest handed 4 GiB to run an idle Node server touches a fraction of it. DAMON is the kernel subsystem that can tell you which fraction, at a cost bounded by region count instead of memory size. Here is how it works, and the three different wrong answers you can get instead.

linuxmemoryinternalsAjay Kumar
11 min read

Per-Sandbox Disk I/O Throttling and the Neighbour You Cannot See

CPU contention is visible and roughly fair. Memory contention has pressure signals. Disk contention shows up as latency in somebody else's unrelated workload, with no attribution and no ticket. Here are the three places you can actually throttle it, what each one really controls, and the two layers that quietly lie to you.

storagecgroupsfirecrackerAjay Kumar

More posts in this topic

165 more, newest first.