CSIPE

Published

- 37 min read

Choosing an Isolation Boundary for AI Agents: Containers vs MicroVMs vs Userspace Sandboxes


The Digital Fortress: Your Everyday Guide to a Safer Digital Life

Stay Safe Online Without Making It Your Second Job

The Digital Fortress (Second Edition)

A warm, plain-English guide for people with real lives and finite patience. Learn the handful of habits that genuinely protect your money, accounts, and family, and get honest permission to ignore the rest.

Buy the book now
The Anonymity Playbook: Digital Survival for Whistleblowers, Journalists, Activists, and Everyone Else

For People Who Cannot Afford to Get Privacy Wrong

The Anonymity Playbook (Second Edition)

A practitioner’s field manual for journalists protecting sources, whistleblowers, and activists. It explains how the surveillance actually works, what each technique costs you, and exactly where it fails.

Buy the book now
Secure Software Development: Practical patterns for building secure software

Write, Ship, and Maintain Code Without Shipping Vulnerabilities

Secure Software Development

A hands-on security guide for developers and IT professionals who ship real software. Build, deploy, and maintain secure systems without slowing down or drowning in theory.

Buy the book now
The Secure Harness: Shipping Production Code with AI Coding Agents

Use AI Coding Agents Without Losing Control of Your Codebase

The Secure Harness

A calm, practical guide to letting agents do useful work inside boundaries you set, enforce, and audit. Ships with 15 copy-pasteable artifacts: hook scripts, permission configs, release gates, and MCP templates.

Buy the book now
The AI Native Engineer: Build, Evaluate, and Ship AI Systems That Work in Production

Stop Shipping Demos. Start Shipping Systems.

The AI Native Engineer

Sixteen hands-on chapters, one real product. Grow it from a single model call into a retrieved, tool-using, observable, production-grade system, with evaluation treated as a habit from the first feature.

Buy the book now

Two requirements that pull apart

Agent workloads want to start instantly and run untrusted code safely. Those requirements have historically traded against each other, which is why the tooling landscape in mid-2026 contains three different answers competing for the same job.

The July 2026 launch of Tarit, a Rust sandbox platform built on rust-vmm, put the tension on display. It boots hardware-virtualised, kernel-isolated environments in milliseconds, and pairs that with warm pools, live snapshots, egress allowlisting, and per-VM terminal access. Every one of those features exists to make strong isolation cheap enough that people will actually use it. Chimera, released the same week, went the other direction: a userspace sandbox that runs untrusted code with no VM, no container, and no special kernel capability.

Both are reasonable engineering. They encode different bets about what you are defending against.

What each boundary actually stops

Clear thinking here starts with the attack that each mechanism is designed to prevent, rather than with the marketing category.

A container shares the host kernel. Namespaces give the process a private view of process IDs, mounts, and networks; cgroups limit resources; seccomp filters which syscalls it may call; capabilities trim root’s powers. What that gets you is protection against a process that misbehaves within the rules. A kernel vulnerability reachable through an allowed syscall gets an attacker to host root, and the kernel exposes hundreds of syscalls with a long history of exploitable bugs. Containers are a strong operational boundary and a moderate security boundary.

A virtual machine gives the guest its own kernel and interposes a hypervisor between guest and host. A guest kernel exploit gets the attacker guest root and no further, because breaking out requires a hypervisor escape, and hypervisor attack surface is orders of magnitude smaller than kernel syscall surface. MicroVMs cut that surface further by shipping a minimal device model, typically a few virtio devices and nothing else. The cost used to be boot time measured in seconds, which is what rust-vmm and its descendants have spent five years reducing.

A userspace sandbox intercepts what the code can do without relying on the kernel to enforce it, using techniques like WebAssembly compilation, syscall interception through ptrace or seccomp-unotify, or a language-level interpreter with a restricted standard library. The isolation quality depends entirely on the completeness of the interception layer, and the failure mode is subtle: a missed syscall or an unsafe capability in the runtime is a silent hole rather than a loud crash.

The layer stack, drawn out

Those three descriptions become easier to compare when you draw the privileged code sitting between the agent process and host root. The amount of that code, and the number of separate interfaces it exposes, carries the whole argument.

WebAssembly

Wasm module

Bounds-checked linear memory

Explicit host imports

Host process

Host kernel

MicroVM

Agent process

Guest kernel
full syscalls, isolated instance

virtio queues

VMM device model
block, net, vsock, serial

KVM ioctl surface

Host kernel

Hardware

gVisor

Agent process

Sentry
syscall table reimplemented in Go

Gofer
file access proxy

Host kernel
filtered syscall subset

Hardware

Hardened container

Agent process

Host Linux kernel
350+ syscalls, every driver

Hardware

The container path has exactly one hop. A syscall from the agent lands in code that runs with full host privilege and that simultaneously serves every other process on the machine. x86-64 Linux exposes more than 350 syscalls, and the reachable surface behind them covers filesystem drivers, the network stack, the eBPF verifier, and the cgroup interface. Most container escapes in the public record are a bug somewhere in that surface, reached through a syscall the default profile happily allows.

gVisor adds a hop and shrinks the far end. Sentry implements the syscall table itself, in Go, and issues only a small filtered set of host syscalls on the guest’s behalf. File access leaves through a separate Gofer process over a dedicated protocol, so the sandboxed application never touches host filesystem code paths directly. A guest that finds a bug in Sentry lands inside a process that is itself confined by seccomp and namespaces, which buys you a second wall rather than an outright win.

The microVM path looks longer and every step along it is narrower than it appears. The guest kernel is a real kernel, so guest privilege escalation is cheap and unremarkable. It buys nothing, because the next interface is a handful of virtio queues serviced by a VMM whose entire device model amounts to a block device, a network device, a vsock channel, a serial console, and a controller used to trigger reset. Firecracker ships that list on purpose. No USB, no floppy controller, no PCI hotplug, no graphics adapter. The classic hypervisor escapes people quote, VENOM in the QEMU floppy controller and the SLIRP heap overflows, targeted devices Firecracker never emulates.

WebAssembly deletes the syscall interface altogether. The module addresses a linear memory the runtime bounds-checks on every access, and calls out only through function imports the embedder wrote by hand. A module with no imports cannot open a file, because the verb does not exist in its vocabulary.

BoundaryFirst privileged interface reachedRough size of that interfaceWhat an escape requires
Hardened containerHost kernel syscalls350+ syscalls minus whatever seccomp deniesOne kernel bug behind an allowed syscall
gVisorSentry, then a filtered host syscall setA Go reimplementation, then roughly 50 host syscallsA Sentry bug plus a host kernel bug in the filtered set
MicroVMvirtio queues into the VMM device modelFour to six emulated devicesA VMM device bug, or a KVM bug
Full VM (QEMU)virtio plus a large legacy device modelDozens of devices, many of them legacyA device emulation bug, of which QEMU has shipped many
WebAssemblyHost functions the embedder importedExactly the list you wroteA runtime codegen bug, or an unsafe import you exposed

Container escape classes and what stops each

Interface sizes are an abstraction. The public vulnerability record is concrete, and it sorts into a small number of classes that behave very differently depending on which boundary you picked.

Escape classPublic exampleHardened containergVisorMicroVMWasm
Container runtime bugCVE-2024-21626, the runc file descriptor leak reachable through WORKDIRNoDifferent runtime, different bugsYes, the host runtime is out of guest reachYes
Kernel privilege escalation via an allowed syscallCVE-2022-0847 (Dirty Pipe), CVE-2021-22555 (netfilter), CVE-2023-0386 (OverlayFS)NoMostly, the host code path is never invokedYes, escalation stops at guest rootNot applicable
Legacy kernel interface abuseCVE-2022-0492, cgroup v1 release_agentOnly if you deny the interfaceYesYesYes
Host-side hook or device plugin injectionCVE-2025-23266 in the NVIDIA Container ToolkitNoNo, the same toolkit runs on the hostPartly, GPU passthrough relocates the problemYes
Deliberate misconfigurationMounted container socket, hostPID, --privileged, host path mountsNoNoNoNo
Microarchitectural side channelSpectre and MDS familiesNoPartialPartial, depends on microcode and schedulingPartial

Two rows deserve a second reading. The misconfiguration row has no yes in it, and that matters more than any of the others, because the most common escape in production is a mount rather than a memory corruption bug. A microVM handed the host’s container socket is escaped the instant it starts. Isolation technology has no opinion about what you voluntarily passed through it, and the strongest boundary on the market cannot un-give a capability you granted at launch.

The hook injection row is the one that catches teams running GPU workloads. Giving a sandbox access to an accelerator drags in a device plugin, a host driver stack, and a container toolkit, all of which execute on the host with privilege before the workload ever starts. NVIDIAScape, CVE-2025-23266, scored 9.0 and worked through OCI hook handling in that toolkit, which means the escape happened in host-side setup code that sat entirely outside whatever boundary the workload was going to run in. A microVM with a passed-through GPU still depends on the host VFIO path and on the same toolkit that arranged the passthrough.

The side channel row is the honest one. Moving from a container to a microVM barely changes your exposure to cross-tenant microarchitectural leakage, because the shared resource being attacked is the physical core and its caches. Mitigations live in microcode, in kernel flush-on-switch settings, and in scheduling policy. Core scheduling, disabling SMT for sensitive pools, or simply not co-tenanting different trust domains on one physical host are the controls that move the needle, and all three cost real money in wasted capacity.

Matching boundary to threat

The decision follows from one question. What is the worst thing the code inside the sandbox might be?

If the code is your own, generated by a model you called, running against your own repository, then the realistic threat is a mistake or a prompt injection that makes the agent do something unintended. A hardened container handles that. The agent has no exploit development capability aimed at your kernel, and the value of adding a hypervisor is small relative to its operational cost.

If the code comes from somewhere you do not control, the calculation changes. Executing a dependency’s install script, running a user-submitted snippet, or evaluating output from a model that read untrusted input all mean genuinely hostile code may execute. A container is thin protection against an attacker who brought a kernel exploit, and the microVM earns its keep.

If you are running multiple tenants on shared hardware, the answer is a VM boundary and the discussion is over. Container escape between tenants is a company-ending incident, and every serious multi-tenant platform reached this conclusion years ago.

A threat model per deployment shape

Those three cases cover the logic, and in practice agent deployments arrive in a slightly larger set of recognisable shapes. Naming the shape usually settles the argument faster than debating the technology.

Deployment shapeWhere the code comes fromTenancyMinimum sensible boundaryThe thing that actually bites
Local dev assistant on an engineer’s laptopModel output against your repoOneHardened container, or the OS sandbox already presentCredential scope, rarely escape
CI agent editing your repo and running your testsModel output plus your entire dependency treeOne per repoHardened container, seccomp, no host socketDependency install scripts
Internal tool sandbox running model-written snippetsModel output shaped by untrusted inputOne organisationgVisorPrompt injection reaching the syscall layer
Customer-facing code interpreterEnd users, directlyManyMicroVMEscape between paying customers
Package or repository evaluation at scaleArbitrary third-party publishersManyMicroVM, one per job, no reuseDeliberate, funded exploitation
Browser agent executing what it readsWeb contentManyMicroVM plus the browser’s own sandboxInjected instructions and drive-by payloads
Regulated on-premise deploymentVariesOneWhatever the auditor’s control set names, usually a VMProducing evidence, not surviving exploitation

The CI row is the one teams consistently misjudge. An agent that adds a dependency has decided to execute a stranger’s code on your build host, because npm install runs postinstall scripts and pip install runs setup.py, both with whatever registry tokens and cloud credentials the build environment happens to carry. That is third-party code execution wearing a friendly name, and it lands in the same threat category as the code interpreter row even though the pipeline looks internal. Treating package installation as a separate, more strongly isolated step than the rest of the agent’s work is a cheap fix that most pipelines can absorb.

The browser row deserves similar suspicion. A browser agent reads attacker-controlled text on every page it visits, and it usually holds session cookies for services the operator cares about. The browser process brings its own strong sandbox, which is worth keeping, and it does nothing about an agent that reads an injected instruction and dutifully calls a tool with it. Isolation caps the damage; the instruction still executes.

No

No

Yes

Yes

Yes

No

No

Yes

No

Yes

No

Yes

Agent needs to execute code

Does code from outside
your organisation run?

Does the sandbox hold
production credentials?

Hardened container
seccomp, read-only, no caps

Hardened container
plus short-lived scoped tokens
and egress allowlist

More than one tenant
on shared hardware?

MicroVM, one per job
no reuse without reset

Does the workload need
a real kernel and toolchain?

Can it compile to Wasm?

Wasm with explicit imports

gVisor

Syscall-heavy or
filesystem-heavy?

Egress allowlist, no ambient credentials,
reset proven by test

Every path in that tree terminates at the same node, which is the point. The boundary decision changes how hard an incident is to cause. The credential and egress decisions change what the incident is worth, and they apply identically whether you picked Wasm or a hypervisor.

The features that make strong isolation usable

Warm pools deserve attention because they are the trick that resolves the original tension. Rather than booting a VM when a request arrives, the platform keeps a pool of booted, idle VMs and assigns one on demand. The user-visible latency becomes assignment time, measured in single-digit milliseconds, while the isolation remains a full hardware boundary.

The correctness requirement is that a VM is never reused across trust domains without a reset. Live snapshots make this cheap: capture the VM state immediately after boot, and restore from that snapshot after each use rather than rebooting. A reused-without-reset VM leaks the previous tenant’s data and is worse than no isolation at all, because the architecture diagram says the boundary is there.

Egress allowlisting matters more for agent workloads than for most compute. The isolation boundary stops the sandbox from reaching the host; it says nothing about the sandbox reaching the internet. An agent that can read a secret from its own working directory and POST it anywhere has exfiltrated it regardless of how strong the VM boundary is. Enforcement belongs outside the sandbox, in the network path, where the sandboxed code cannot modify it.

Per-VM terminal access is an operational feature with a security consequence worth noting. Being able to attach to a running sandbox is invaluable for debugging and is also a channel into the isolated environment, so it needs the same authentication and audit as any other production access path.

Building a warm pool with snapshot restore

Those four features are easy to describe and easy to get wrong, so the pool implementation is worth walking through step by step. The shape below assumes Firecracker, though Cloud Hypervisor’s snapshot and restore flow follows the same sequence with different endpoint names.

Start by building a template. You need an uncompressed kernel image, a read-only root filesystem containing the agent’s toolchain, and an init that starts your supervisor and signals readiness over vsock. Boot that template once, wait for the readiness signal, pause the VM, and capture a full snapshot: one memory file and one VM state file.

   # 1. Boot the template, wait for readiness, then freeze it.
curl --unix-socket /run/fc-template.sock -X PATCH 'http://localhost/vm' \
  -H 'Content-Type: application/json' \
  -d '{"state": "Paused"}'

# 2. Capture memory and device state. This pair is the pool's seed.
curl --unix-socket /run/fc-template.sock -X PUT 'http://localhost/snapshot/create' \
  -H 'Content-Type: application/json' \
  -d '{
        "snapshot_type": "Full",
        "snapshot_path": "/srv/pool/agent-v7.snap",
        "mem_file_path": "/srv/pool/agent-v7.mem"
      }'

Serve the memory file through a userfaultfd handler rather than copying it per restore. Pages then map lazily on first touch, so a thousand restores from a 512 MB template do not cost 512 GB of page cache. Each restore gets a freshly created overlay disk and a freshly created tap device with a new MAC address, and only then loads the snapshot and resumes.

   # 3. Per job: fresh overlay, fresh tap, then restore.
truncate -s 4G "/run/jobs/${JOB}/overlay.ext4"
ip tuntap add "tap-${JOB}" mode tap && ip link set "tap-${JOB}" up

curl --unix-socket "/run/jobs/${JOB}/fc.sock" -X PUT 'http://localhost/snapshot/load' \
  -H 'Content-Type: application/json' \
  -d '{
        "snapshot_path": "/srv/pool/agent-v7.snap",
        "mem_backend": { "backend_type": "Uffd", "backend_path": "/run/uffd.sock" },
        "enable_diff_snapshots": false,
        "resume_vm": true
      }'

never

Template VM booted once

Pause and snapshot
mem file + state file

Snapshot on disk
served over userfaultfd

Restore N instances
fresh overlay, fresh tap, new MAC

Warm, idle, unassigned

Assign to job
inject scoped credentials after resume

Job runs

Kill VMM, delete overlay,
delete tap, flush conntrack

Second job on the same instance

The reset requirement decomposes into six pieces of state, and skipping any one of them produces a boundary that exists on the diagram and nowhere else.

  • Memory. Always restore from the snapshot. A used instance never returns to the warm pool, no matter how clean it looks.
  • Disk. The per-job overlay is destroyed, and the template disk stays read-only so nothing from a previous job can have reached it.
  • Network identity. Fresh tap, fresh MAC, fresh address, and a conntrack flush for the old address so the next tenant does not inherit established flows.
  • Entropy. Every instance restored from the same snapshot resumes with an identical CSPRNG state, so two jobs can generate the same “random” session token, temp file name, or TLS nonce. Firecracker’s documentation calls this out directly. Attach a virtio-rng device and reseed the guest immediately on resume, before workload code runs.
  • Clock. The guest wakes believing the snapshot’s wall clock. Sync it on resume or certificate validation and token expiry will misbehave in ways that look like flakiness.
  • Credentials. Per-job secrets are injected after resume through vsock or a metadata service, never baked into the template.

The entropy item is the one that reliably surprises people, because nothing crashes. Two tenants quietly receive the same identifier and the failure surfaces weeks later as an inexplicable collision in a database.

Prove the reset with a test rather than a code review. Write a canary file and a random value to a warm instance, return it to the pool through whatever path production uses, request a new instance, and assert both are absent. Run it on every deploy of the pool manager.

The options between the extremes

Presenting this as three choices is a simplification, and two intermediate technologies are worth knowing because they often fit agent workloads better than either pure option.

Kernel-emulating sandboxes such as gVisor sit between containers and VMs. They intercept guest syscalls in a userspace kernel written in a memory-safe language, so the host kernel sees a small, controlled set of calls rather than the full syscall surface. The isolation is meaningfully stronger than a plain container and the startup cost stays close to container speed. The trade is compatibility and performance: syscall-heavy workloads slow down noticeably, and applications relying on unusual kernel features may not run at all. For an agent doing file edits and HTTP calls, neither limitation bites hard.

WebAssembly runtimes go further in the other direction. Compiling the workload to Wasm gives you a boundary enforced by the module format itself, with no ambient authority: the code can do nothing except what the host explicitly imports into it. Startup is measured in microseconds. The catch is that the workload has to be compilable to Wasm and content with a restricted view of the world, which rules out running arbitrary developer toolchains but suits a constrained tool-execution sandbox very well.

A pattern worth considering is mixing these by consequence rather than picking one. Tool calls that only transform data run in Wasm, where startup is free and the boundary is total. Tool calls that need a real filesystem and real binaries get a gVisor container. Anything executing genuinely untrusted third-party code gets a microVM. The routing logic is a small amount of code and the aggregate cost drops sharply compared to putting everything in the strongest available box.

Five options side by side

Routing by consequence requires knowing what each route costs, so here is the full set with the numbers that drive the decision. Treat every figure as an order of magnitude rather than a promise, because configuration moves all of them.

Hardened containergVisorMicroVM (Firecracker, Cloud Hypervisor)Full VM (Kata with QEMU)Wasm (Wasmtime)
Host-privileged interfaceFull syscall tableSentry, then filtered syscallsvirtio plus KVMvirtio, legacy devices, KVMHost imports you wrote
Cold startTens to low hundreds of msComparable to a container, plus Sentry initFirecracker documents under 125 ms to userspace; single-digit ms from snapshotHundreds of ms to secondsMicroseconds
Memory floor per instanceTens of MBExtra tens of MB for Sentry and GoferFirecracker documents roughly 5 MB VMM overhead, plus the guest kernel100 MB and upKilobytes to a few MB
CompatibilityAnything that runs on LinuxMost things; some /proc and /sys entries and unusual syscalls are missingAnything the guest kernel supportsAnythingOnly what compiles to Wasm
Syscall-heavy throughputNativeThird-party benchmarks report roughly 10 to 40 percent slowerClose to nativeClose to nativeNot applicable
Filesystem-heavy throughputNativeThe weakest point, due to the Sentry-to-Gofer hopNear native over virtio-blkNear nativeHost-mediated
GPU accessYes, via device pluginLimitedPassthrough, awkwardPassthroughNo
Operational costYou already run itA new runtime binary, a node pool, a class of “works everywhere but here” bugsA kernel image, a rootfs image, a VMM, a snapshot format, and a patch cadence for all fourThe same, with a heavier device modelA new build target and a host you maintain

The performance rows split by workload rather than by boundary, which is why generic benchmark headlines mislead. A compute-bound job that runs in one process and touches few files barely notices gVisor. A build that forks thousands of processes and stats hundreds of thousands of paths notices immediately, and published measurements of filesystem-heavy work under gVisor land far worse than the syscall figures suggest. gVisor’s directfs and in-memory overlay options recover a useful chunk of that, which makes measuring your own image mandatory before you accept or reject the runtime on someone else’s numbers.

MicroVM numbers look better than gVisor’s in almost every row, and the operational cost row is where the trade actually lands. Adopting Firecracker means you now own a kernel image and its CVE feed, a root filesystem and its package updates, a VMM binary, and a snapshot format that has to stay compatible across your own deploys. It also means bare metal instances or nested virtualisation, since KVM needs hardware access. Nested virtualisation is available on the major clouds and runs slower; bare metal comes in large units at a higher price per core. The isolation decision has an invoice attached, and the invoice is usually the reason teams settle on gVisor for the middle tier of their workloads.

The Wasm column is unbeatable everywhere except compatibility, and compatibility is the column that decides whether the option exists at all. If the sandboxed work is data transformation, expression evaluation, or a policy check, Wasm wins outright. If it needs to run git, a package manager, or a language toolchain, the column is empty and the discussion moves elsewhere.

Configuring a microVM without surprises

Picking Firecracker is the easy part. The configuration below is the part that determines whether the boundary you bought is the boundary you get.

   {
	"boot-source": {
		"kernel_image_path": "/srv/pool/vmlinux-6.12",
		"boot_args": "console=ttyS0 reboot=k panic=1 pci=off nomodule init=/sbin/agent-init"
	},
	"machine-config": {
		"vcpu_count": 2,
		"mem_size_mib": 2048,
		"smt": false
	},
	"drives": [
		{
			"drive_id": "rootfs",
			"path_on_host": "/srv/pool/rootfs.ext4",
			"is_root_device": true,
			"is_read_only": true
		},
		{
			"drive_id": "scratch",
			"path_on_host": "/run/jobs/JOB/overlay.ext4",
			"is_root_device": false,
			"is_read_only": false,
			"rate_limiter": { "bandwidth": { "size": 104857600, "refill_time": 1000 } }
		}
	],
	"network-interfaces": [
		{
			"iface_id": "eth0",
			"host_dev_name": "tap-JOB",
			"guest_mac": "AA:FC:00:00:07:19",
			"rx_rate_limiter": { "bandwidth": { "size": 12500000, "refill_time": 1000 } },
			"tx_rate_limiter": { "bandwidth": { "size": 12500000, "refill_time": 1000 } }
		}
	],
	"entropy": {},
	"vsock": { "guest_cid": 3, "uds_path": "/run/jobs/JOB/vsock.sock" }
}

smt: false removes the sibling hyperthread as a side channel path between co-scheduled guests, at the obvious cost in density. The read-only root device with a separate writable scratch drive is what makes the template immutable across the pool. Rate limiters convert a noisy-neighbour problem into a bounded one, and they double as a detection signal: a job pinned against its network ceiling for ten minutes is doing something worth reading the logs about. Attach virtio-rng through the entropy stanza, since that device is the reseeding path the snapshot flow depends on.

Then run it under the jailer, always. The jailer chroots the VMM, drops it to an unprivileged uid and gid, places it in its own namespaces and cgroup, and lets Firecracker install seccomp filters on its own threads. One VMM process per VM means a compromised device model is confined to one tenant’s blast radius.

   jailer \
  --id "$JOB" \
  --exec-file /usr/bin/firecracker \
  --uid 30000 --gid 30000 \
  --chroot-base-dir /srv/jail \
  --cgroup-version 2 \
  --cgroup "memory.max=2684354560" \
  --cgroup "pids.max=2048" \
  -- --config-file /vm.json --no-api

Cloud Hypervisor is the right pick when the microVM needs things Firecracker deliberately omits: PCI, VFIO device passthrough for accelerators, memory hotplug, or a virtiofs mount. It carries a larger device model in exchange, which is a real trade rather than a free upgrade, and it keeps its own seccomp enforcement on by default.

   cloud-hypervisor \
  --api-socket "/run/jobs/$JOB/ch.sock" \
  --cpus boot=2 \
  --memory size=2048M \
  --kernel /srv/pool/vmlinux-6.12 \
  --cmdline "console=hvc0 root=/dev/vda1 ro" \
  --disk path=/srv/pool/rootfs.raw,readonly=on path="/run/jobs/$JOB/overlay.raw" \
  --net "tap=tap-$JOB,mac=$MAC" \
  --rng src=/dev/urandom \
  --seccomp true \
  --console off

gVisor where it fits

Between the container baseline and the microVM sits the runtime most teams end up defaulting to, and it needs a little tuning before it behaves well.

Install runsc, register it with containerd as a runtime handler, and point it at a config file rather than scattering flags across the node.

   # /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]
    TypeUrl = "io.containerd.runsc.v1.options"
    ConfigPath = "/etc/containerd/runsc.toml"
   # /etc/containerd/runsc.toml
[runsc_config]
  platform = "systrap"        # default since 2023; faster than the old ptrace platform
  network = "sandbox"         # netstack in userspace, not the host network stack
  overlay2 = "root:memory"    # writes land in a tmpfs overlay, discarded on exit
  directfs = "true"           # cuts Gofer round trips for file operations
  watchdog-action = "panic"   # a stuck sandbox dies loudly instead of hanging

Four of those five lines are performance or containment settings that ship off or weaker by default in some distributions, so check rather than assume. network = "sandbox" matters most for the security story: gVisor’s own userspace network stack handles the guest’s traffic, so a bug in the guest’s networking never reaches the host’s stack. Setting it to host recovers throughput and gives back a large piece of what you paid for.

The Sentry itself runs under a seccomp filter, which is the property that makes gVisor more than a syscall translator. An attacker who achieves code execution inside Sentry is executing inside a Go process confined to a small host syscall set, in its own namespaces, with no ambient filesystem access. Two independent bugs are needed for a host compromise.

What breaks is the part to budget for. gVisor implements a large fraction of the Linux syscall surface and not all of it, so unusual calls return ENOSYS, some /proc and /sys entries are absent or simplified, io_uring support is limited, and workloads depending on FUSE or on loading kernel modules are out. None of that is exotic in the abstract and all of it is specific to your image, which makes the only useful test running your actual agent container under runsc and reading the logs. Budget a day. Most agent images that edit files and make HTTP calls pass on the first attempt; images carrying a full build toolchain frequently need a fix or two.

A Wasmtime host that grants nothing by default

The Wasm option inverts the configuration problem. Rather than removing capabilities from something that starts with all of them, you write down the complete list of things the guest may do, in a language you can review in a pull request.

   use wasmtime::{Config, Engine, Linker, Module, Store, StoreLimits, StoreLimitsBuilder};
use wasmtime_wasi::preview1::{self, WasiP1Ctx};
use wasmtime_wasi::{DirPerms, FilePerms, WasiCtxBuilder};

struct HostState {
    wasi: WasiP1Ctx,
    limits: StoreLimits,
}

fn run_task(wasm_path: &str, workspace: &str) -> anyhow::Result<()> {
    let mut config = Config::new();
    config.consume_fuel(true);          // bound total instructions executed
    config.epoch_interruption(true);    // bound wall-clock time independently
    let engine = Engine::new(&config)?;

    // One preopened directory. No environment, no argv, no sockets.
    let wasi = WasiCtxBuilder::new()
        .preopened_dir(workspace, "/workspace", DirPerms::all(), FilePerms::all())?
        .build_p1();

    let limits = StoreLimitsBuilder::new()
        .memory_size(256 << 20)         // hard ceiling on linear memory growth
        .instances(1)
        .build();

    let mut store = Store::new(&engine, HostState { wasi, limits });
    store.limiter(|s| &mut s.limits);
    store.set_fuel(5_000_000_000)?;
    store.set_epoch_deadline(1);

    let mut linker: Linker<HostState> = Linker::new(&engine);
    preview1::add_to_linker_sync(&mut linker, |s| &mut s.wasi)?;

    // The entire capability surface beyond WASI files: one reviewed function.
    linker.func_wrap(
        "agent",
        "emit_result",
        |mut caller: wasmtime::Caller<'_, HostState>, ptr: u32, len: u32| -> anyhow::Result<()> {
            // Copy out of guest linear memory, validate length and encoding,
            // forward to the caller's result channel. Nothing else is exposed.
            let _ = (&mut caller, ptr, len);
            Ok(())
        },
    )?;

    let module = Module::from_file(&engine, wasm_path)?;
    let instance = linker.instantiate(&mut store, &module)?;
    instance
        .get_typed_func::<(), ()>(&mut store, "run")?
        .call(&mut store, ())?;
    Ok(())
}

Exact signatures shift between Wasmtime releases, so treat the shape as the lesson rather than the code as copy-paste. Four properties carry the security value. Fuel bounds total work, so a module that loops forever runs out rather than pinning a core. Epoch interruption bounds wall-clock time independently, which catches the case where a module blocks rather than spins. The store limiter caps memory growth at a number you chose instead of at whatever the host has left. One host function sits in the linker, and it is the complete list of things the guest can reach outside its own memory.

The failure mode to watch for is capability creep in review. Someone needs an HTTP call, wasi:http gets added, and the sandbox now speaks to the network. Someone needs a socket, wasi:sockets gets added, and the egress story reverts to whatever the host process can reach. Each addition is individually reasonable and the aggregate is a sandbox with an undocumented perimeter. Keep the import list in one file, require a security review to change it, and diff it in CI.

One clarification worth stating plainly, since it trips people up: the Wasm boundary protects the host from the module and says nothing about the module’s internal memory safety. A buffer overflow inside a C program compiled to Wasm still corrupts that program’s own linear memory and still produces wrong answers. The corruption simply cannot reach past the sandbox wall.

Checking that the boundary is real

An isolation boundary you have not tested is an assumption. Three checks catch most misconfigurations and take an afternoon.

Try to reach the host from inside. Attempt to read a file that should be outside the sandbox, list host processes, and connect to a host-only service. Every one of those should fail in a way you can see in a log, and a surprising number of production sandboxes fail this test because a debugging convenience was added and never removed.

Try to reach the network. From inside the sandbox, connect to an address that is definitely not on the allowlist and confirm the connection is refused rather than merely slow. Then confirm that DNS resolution for that address also fails, since a sandbox that can resolve arbitrary names has a low-bandwidth exfiltration channel even with TCP blocked.

Try to persist. Write a file, then destroy and recreate the sandbox, and confirm the file is gone. This is the check that catches warm-pool reuse without reset, and it is the one most worth automating, because the failure appears only under the specific conditions of pooled reuse and will not show up in a manual test of a freshly booted instance.

   # Minimal escape-attempt suite, run from inside the sandbox.
# Every line should fail. Any success is a finding.
cat /etc/shadow                        2>&1 | head -1
ls /proc/1/root                        2>&1 | head -1
curl -sS --max-time 5 https://example.org/ 2>&1 | head -1
getent hosts example.org               2>&1 | head -1
mount -t tmpfs none /mnt               2>&1 | head -1
unshare -Ur id                         2>&1 | head -1

Building versus buying

An analysis circulating on the infrastructure side in July 2026 listed five layers that production agent systems need: warm browser pools, VM-level isolation, coherent identities with residential routing, unified replay and traces, and multi-model gateways. Its conclusion was that building internally makes sense when the infrastructure is a strategic differentiator or a hard requirement, and otherwise does not.

That conclusion holds up. Each of those layers is a real engineering effort, and the isolation layer in particular is one where mistakes are quiet. A container that should have been a VM works perfectly in every test you write, right up until someone brings a kernel exploit.

The exception is compliance-driven work. Regulated environments frequently need the workload inside a specific network boundary or jurisdiction, and vendor platforms may not offer that. Building becomes the only option, and the honest budget includes the ongoing cost of patching a hypervisor and a guest kernel image, not just the initial implementation.

A configuration worth copying

Whatever boundary you choose, a few settings apply across all three and cost nothing.

   # Container baseline for agent execution. Every flag removes a known path.
docker run --rm \
  --user 1000:1000 \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=512m \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --security-opt seccomp=./agent-seccomp.json \
  --pids-limit 512 \
  --memory 4g \
  --network agent-egress-restricted \
  -v "$PWD/workspace:/workspace:rw" \
  agent-image

--read-only with a noexec tmpfs is the pairing worth explaining. Together they mean the agent cannot write a file anywhere and then execute it, which breaks the standard pattern of downloading a payload and running it. Builds that need to write executables will fail, and the fix is a writable mount scoped to the build output directory rather than removing the flag.

--pids-limit and --memory are denial-of-service controls that also serve as detection. An agent that suddenly hits its process limit is doing something worth looking at.

Writing a seccomp profile you can defend

That command references a profile file, and the file deserves its own treatment, because the vendor defaults answer a different question than yours. Docker’s default profile allows the large majority of the syscall table and denies a few dozen known-dangerous calls, which suits general workloads and gets the polarity backwards for an agent sandbox. Start from deny and write down what your agent needs.

   {
	"defaultAction": "SCMP_ACT_ERRNO",
	"defaultErrnoRet": 1,
	"architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_AARCH64"],
	"syscalls": [
		{
			"comment": "Ordinary process, memory and file work. None of it reaches host state.",
			"action": "SCMP_ACT_ALLOW",
			"names": [
				"read",
				"write",
				"readv",
				"writev",
				"pread64",
				"pwrite64",
				"openat",
				"close",
				"lseek",
				"fstat",
				"newfstatat",
				"statx",
				"ftruncate",
				"getdents64",
				"getcwd",
				"chdir",
				"fchdir",
				"renameat2",
				"mkdirat",
				"unlinkat",
				"symlinkat",
				"readlinkat",
				"fchmodat",
				"faccessat2",
				"mmap",
				"mprotect",
				"munmap",
				"mremap",
				"madvise",
				"brk",
				"rt_sigaction",
				"rt_sigprocmask",
				"rt_sigreturn",
				"sigaltstack",
				"futex",
				"set_robust_list",
				"get_robust_list",
				"rseq",
				"set_tid_address",
				"clock_gettime",
				"clock_nanosleep",
				"nanosleep",
				"getrandom",
				"getpid",
				"gettid",
				"tgkill",
				"wait4",
				"execve",
				"execveat",
				"exit",
				"exit_group",
				"prlimit64",
				"sched_getaffinity",
				"sched_yield",
				"uname",
				"fcntl",
				"ioctl",
				"dup3",
				"pipe2",
				"eventfd2",
				"memfd_create",
				"epoll_create1",
				"epoll_ctl",
				"epoll_pwait",
				"ppoll"
			]
		},
		{
			"comment": "IPv4 sockets only. AF_PACKET, AF_NETLINK and AF_VSOCK fall through to the default deny.",
			"action": "SCMP_ACT_ALLOW",
			"names": ["socket"],
			"args": [{ "index": 0, "value": 2, "op": "SCMP_CMP_EQ" }]
		},
		{
			"comment": "IPv6 sockets, same reasoning.",
			"action": "SCMP_ACT_ALLOW",
			"names": ["socket"],
			"args": [{ "index": 0, "value": 10, "op": "SCMP_CMP_EQ" }]
		},
		{
			"action": "SCMP_ACT_ALLOW",
			"names": [
				"connect",
				"sendto",
				"recvfrom",
				"sendmsg",
				"recvmsg",
				"setsockopt",
				"getsockopt",
				"getsockname",
				"getpeername",
				"shutdown"
			]
		},
		{
			"comment": "Threads yes, new namespaces no. The mask covers the CLONE_NEW* flags.",
			"action": "SCMP_ACT_ALLOW",
			"names": ["clone"],
			"args": [{ "index": 0, "value": 2114060288, "valueTwo": 0, "op": "SCMP_CMP_MASKED_EQ" }]
		},
		{
			"comment": "clone3 hides its flags in a struct, so libseccomp cannot filter them. Returning ENOSYS makes glibc fall back to clone, which is filterable.",
			"action": "SCMP_ACT_ERRNO",
			"errnoRet": 38,
			"names": ["clone3"]
		},
		{
			"comment": "Loud denials. Each entry is a documented escape path or a seccomp bypass.",
			"action": "SCMP_ACT_KILL_PROCESS",
			"names": [
				"mount",
				"umount2",
				"pivot_root",
				"chroot",
				"unshare",
				"setns",
				"bpf",
				"perf_event_open",
				"keyctl",
				"add_key",
				"request_key",
				"ptrace",
				"process_vm_readv",
				"process_vm_writev",
				"pidfd_getfd",
				"name_to_handle_at",
				"open_by_handle_at",
				"userfaultfd",
				"init_module",
				"finit_module",
				"delete_module",
				"kexec_load",
				"kexec_file_load",
				"reboot",
				"swapon",
				"swapoff",
				"io_uring_setup",
				"io_uring_enter",
				"io_uring_register",
				"acct",
				"quotactl",
				"syslog",
				"personality"
			]
		}
	]
}

Four choices in there carry most of the value. The default action returns EPERM rather than killing, because a denied call that surfaces as an ordinary error is far easier to diagnose than a process that vanishes. Explicit denials use SCMP_ACT_KILL_PROCESS instead, since nothing legitimate in an agent image calls init_module, and a workload that tries has already gone wrong in a way you want recorded loudly.

The clone3 entry looks like a hack and earns its place. Argument filtering works on register values, and clone3 passes a pointer to a struct, so a filter cannot read the flags without a TOCTOU race. Returning ENOSYS pushes glibc onto the legacy clone path where the flag word sits in a register and the mask above applies. Profiles that forget this let a workload create a user namespace through clone3 while the author believes namespaces are blocked.

io_uring belongs in the denial list for a reason worth stating explicitly. Operations submitted through an io_uring ring are performed by kernel worker threads and do not pass through the calling task’s seccomp filter, so a profile that denies openat while allowing io_uring_setup denies nothing at all. Deny the three ring syscalls unless you have measured a need and accepted the consequence.

Build the allowlist by recording rather than guessing. Run the image with defaultAction set to SCMP_ACT_LOG in a staging environment, exercise the real workload including its error paths, and collect the SECCOMP audit records. On Kubernetes the Security Profiles Operator does the same job with an eBPF recorder and writes the profile object for you. Then perform the step everyone skips: remove entries. A profile that only ever grows has stopped being a control.

Kubernetes as the deployment surface

Most of this ends up on Kubernetes eventually, and the cluster adds both the mechanism for selecting a runtime and several ways to hand the boundary back. RuntimeClass is the selector.

   apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    sandbox.example.com/runtime: gvisor
  tolerations:
    - key: sandbox.example.com/dedicated
      operator: Equal
      value: agent
      effect: NoSchedule
overhead:
  podFixed:
    memory: '128Mi'
    cpu: '100m'
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata-clh
handler: kata-clh
scheduling:
  nodeSelector:
    sandbox.example.com/runtime: kata
overhead:
  podFixed:
    memory: '192Mi'
    cpu: '250m'

overhead.podFixed is the field teams forget. Sandbox runtimes consume memory outside the container’s own accounting, and without the declaration the scheduler believes a node that is genuinely full still has room. The symptom is node-level memory pressure that evictions blame on the wrong pod.

RuntimeClass selects a runtime and enforces nothing. Anything able to create a pod can omit runtimeClassName and land on the default runtime, which means the control belongs in admission policy rather than in the workload manifest. Require the sandboxed class for every pod in the agent namespace with a validating policy, and taint the sandbox node pool so those pods cannot land anywhere else.

   apiVersion: v1
kind: Pod
metadata:
  name: agent-job-7f21
  namespace: agent-sandbox
spec:
  runtimeClassName: gvisor
  automountServiceAccountToken: false
  enableServiceLinks: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 65532
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/agent-deny-by-default.json
  containers:
    - name: agent
      image: registry.internal/agent@sha256:0f1e2d...
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: { drop: ['ALL'] }
      resources:
        limits: { cpu: '2', memory: 4Gi, ephemeral-storage: 2Gi }
      volumeMounts:
        - { name: workspace, mountPath: /workspace }
        - { name: tmp, mountPath: /tmp }
  volumes:
    - name: workspace
      emptyDir: { sizeLimit: 2Gi }
    - name: tmp
      emptyDir: { medium: Memory, sizeLimit: 512Mi }

automountServiceAccountToken: false earns its line. A sandbox pod with a projected token and network reach to the API server is an escape into the cluster requiring no kernel bug whatsoever, and the default is to mount one. Set Pod Security Admission to restricted on the namespace as the floor underneath all of this, which blocks privileged containers, host namespaces, and host path mounts without anyone having to remember.

Egress needs the same treatment as everything else in the cluster, and NetworkPolicy alone will not deliver it. Policy matches on IP addresses and CIDR blocks, so allowing a CDN range allows every customer on that CDN, and allowing a package registry allows every package anyone has ever published to it. Send everything through an egress proxy instead, and write the policy so the sandbox can reach the proxy and nothing else.

   apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-egress-proxy-only
  namespace: agent-sandbox
spec:
  podSelector: {}
  policyTypes: ['Egress']
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: egress }
          podSelector:
            matchLabels: { app: egress-proxy }
      ports:
        - { protocol: TCP, port: 3128 }

Cluster DNS is absent from that policy on purpose. Sandbox pods resolve nothing themselves, the proxy resolves on their behalf against its own allowlist, and the DNS exfiltration channel closes. Keep the agent control plane on a different node pool from the sandboxes, and keep GPU nodes on a third, since the accelerator toolkit runs host-privileged setup code that belongs in its own blast radius.

Anti-patterns that keep showing up

Reviewing agent platforms turns up the same handful of mistakes, and none of them are exotic. Each one takes a good boundary decision and quietly undoes it.

Reusing a warm instance without a reset. Someone notices that restoring a snapshot costs 40 ms and returning an idle instance to the pool costs nothing, and the pool manager grows a fast path. The next tenant inherits the previous tenant’s memory, temp files, and open sessions. Whatever the fast path saves, it cannot be worth the incident, and the only reliable defence is the automated canary test described earlier running on every deploy.

Reaching for --privileged to make a build work. The trigger is usually FUSE, overlayfs, or a nested container daemon. --privileged grants every capability, disables the seccomp and AppArmor profiles, and exposes host devices, so the fix costs more than the problem. Add --device /dev/fuse with SYS_ADMIN scoped through a specific profile, switch to rootless BuildKit, or accept that this workload wanted a VM.

Mounting the container socket so the agent can build images. Access to /var/run/docker.sock or the containerd socket is equivalent to host root, because the agent can request a privileged container with the host filesystem mounted. No isolation technology in this article survives it. Give the agent a remote build service with an authenticated API instead.

Treating egress as a follow-up ticket. Common shapes: an allowlist of *.amazonaws.com that covers every S3 bucket on earth, an allowance for the package registry that serves whatever an attacker published yesterday, and an allowance for raw file hosting on a code forge that serves any file any account uploaded. Allowlist exact hostnames, and where the protocol permits it, exact paths.

Enforcing policy inside the sandbox. A tool allowlist implemented in the agent’s own process, a Python import hook, a shell wrapper that rejects certain commands, a system prompt saying which files are off limits. Code running inside the sandbox can rewrite all of these, so they belong on the outside of the boundary where the sandboxed process cannot touch them. In-sandbox checks make good ergonomics and terrible controls.

Ratcheting the seccomp profile open. Every crash report adds a syscall, nothing is ever removed, and eighteen months later the profile permits ptrace because someone once ran a debugger. Re-record the profile whenever the base image changes materially and diff it against the previous version in review.

Letting sandboxes live for a whole session. A long-lived sandbox accumulates state, and state is what a prompt injection needs to carry an effect from one task into the next. Per-task instances bound what a single successful injection can reach, and warm pools make them affordable.

Calling the boundary tested because the happy path works. Every sandbox works fine when nothing attacks it. Run the escape suite from the earlier section on a schedule against production configuration, and treat a change in its output as a deploy-blocking failure.

The part that is not about isolation

Isolation technology answers one question well: what happens when code inside the box turns hostile. It answers nothing about what the box was given access to in the first place.

A perfectly isolated microVM holding a production database credential and permitted to reach that database is a fully effective attack platform, and the hypervisor will faithfully protect the host while the damage happens elsewhere. Teams that spend their entire security budget selecting an isolation technology, and then mount the credentials directory and allow unrestricted egress, have bought a strong door for a room with an open window.

Pick the boundary that matches your threat, configure it with the flags above, and then spend the remaining effort on the two questions that determine actual blast radius: what credentials exist inside the sandbox, and where is it allowed to connect. Those answers change what an incident costs. The isolation choice mostly changes how hard the incident was to cause.