Inference became a distributed, stateful system. Minnesväv is its reliability layer. When a GPU node dies forty minutes into an agent run, the session re-weaves onto healthy hardware in seconds — no failed request, no re-paid prefill, no lost work.
// design targets, enforced as release gates in our public chaos suite
Disaggregated serving, tiered KV caches, and hour-long agent runs turned every request into a multi-node, stateful pipeline. The stack moves bytes brilliantly — and defines nothing about what happens when a machine fails.
A preemption or OOM destroys the entire accumulated context of an agent run. The request fails, the prefill is re-paid from zero, and forty minutes of compute evaporates.
TTFT spikes from cache eviction, tier thrash, and fragmentation — causes no app-layer trace or GPU counter can see. Diagnosis runs on vibes.
Preemptible capacity is 50–70% cheaper and unusable for stateful serving. Everyone over-pays for on-demand because failure means starting over.
Append-only session state — KV blocks, generation cursor, sampler state — streams continuously to a buddy node over NIXL. Kill the machine; the session re-attaches and keeps decoding from the last committed token.
Every TTFT spike attributed to its real cause — eviction storms, tier misses, transfer stalls, fragmentation — joined per session, between the app traces and the GPU counters where nothing can see today.
A behavioral engine learns who queries which documents and when — pre-warming context off-peak, inheriting document ACLs into the cache, and invalidating surgically when sources change.
Inference state is append-only — strictly simpler to replicate than training state. We exploit that.
A per-node daemon streams finalized KV blocks and generation state to a topology-aware buddy over NVLink or RDMA — zero-copy, paced under 5% of link capacity, invisible to the engine.
Gossip heartbeats plus control-plane leases discover failure in under a second. Cloud preemption notices trigger graceful drain — planned handoff, effectively zero loss.
The session is fenced, the replica rehydrates into a healthy engine, the block table re-maps, and decoding continues mid-stream. Your client's connection never even closes.
Ninety-minute autonomous runs with 100K+ tokens of context die on a single node fault. Minnesväv makes the run continue — and gives your framework hooks for tool-call reconciliation.
Session-level continuity and state-layer diagnosis turn p99 mysteries into attributed causes — and failed-request refunds into a line item you retire.
Preemption becomes a drain-and-migrate, not an outage. Offer reliable spot inference as a new SKU — priced above raw spot, still far below on-demand.
Document caches inherit your DMS permissions, invalidate when documents change, and never leave your perimeter. Production behaves like the demo — at 5pm on month-end.
The resumability plugin for vLLM ships free and open source, with the chaos suite we test it with. Run the kill-and-recover demo on your own fleet in an afternoon — then decide if the platform is worth talking about.
Session resumability for vLLM in a single failure domain, the resume-correctness oracle, and the chaos CLI. Apache-2.0. Run it on anything.
Get the pluginCross-domain protection, graceful spot drain, state-layer observability with latency attribution, multi-tenant fencing and encryption, predictive cache orchestration, SLAs.
Talk to usLaunch a reliable spot inference tier on your existing capacity. We integrate, you price the SKU, we share the uplift. Zero engineering lift on your side.
Partner with usWe're onboarding ten design partners — agent platforms, inference providers, and GPU clouds — ahead of the open-source release. Bring a fleet and a failure story.