New release: MiniMax M3, the Kimi K3 architecture and NVLink fan-out

Every model family lays its KV cache out differently, and every new layout is a new chance for a restore to hand the engine the wrong bytes. So a model joins the supported list only after it passes the same regression on hardware: the pages the engine wrote are compared byte for byte after they come back from the accelerator, the engine is restarted under load, and concurrent agent traffic runs against it.

MiniMax M3 pairs a mixture-of-experts model with MiniMax's sparse attention, which keeps an indexer next to the keys and values: 117 cache tensors on every GPU, all of them now restored. We run the official MXFP8 checkpoint. A 142,665-token prompt that takes 18.3 seconds to recompute comes back from the accelerator in 3.8 seconds after an engine restart, and generation speed under concurrent agents stays level with vLLM without the connector. We recommend vLLM 0.28.0 for MiniMax M3.

Kimi K3 is a hybrid: 69 Kimi Delta Attention layers carry a recurrent state and 24 MLA layers carry keys and values, with cache pages of 864 KiB, more than three times what the accelerator accepts in one object. The connector now describes that topology and fragments those pages. We recommend vLLM 0.29.0 for the Kimi family.

With MLA, which GLM-5.x and DeepSeek use, the latent KV page is identical on every tensor-parallel rank. Until now every GPU copied the whole restored layer out of host memory on its own: on eight GPUs, eight copies of the same bytes through the same host path.

With nvlink_fanout enabled, each GPU copies only its share of a layer's pages, and the GPUs of an NVLink domain exchange the rest over NVLink with one collective per layer. The connector finds the NVLink domains itself and all ranks agree on them before switching the feature on. If any rank cannot take part, it stays off for all of them with a single warning, and restores run exactly as before. A restore that fails on one GPU counts as failed on all of them, so no rank ever computes on a partially filled page.

This is an NVIDIA-only feature. It relies on NCCL over NVLink and on NVML to discover the topology, so it needs CUDA and NVIDIA GPUs joined by NVLink. It applies to MLA models with tensor parallelism of two or more. Its exchange buffers live outside gpu_memory_utilization: about 924 MiB per GPU on GLM-5.3.

At 32 concurrent agents the median generation rate per agent rises from 23.6 to 26.9 tokens per second, +14%. The copy into GPU memory competes with the model's own work, so the more agents run at once, the more the fan-out gives back. Extending the trend from 16 to 32 agents, we expect the gap to keep widening, to about +28% at 64 agents; that point is a projection, drawn dashed, and not yet a measurement.

On a restore from the accelerator after an engine restart, time to first token drops from 2.11 to 2.04 seconds, −3%.

The comparison that matters most for agents is the plain one: the same vLLM build, model and GPUs, with the ADP connector and without it. Every coding agent below carries a 33,000 to 40,000-token history, and the histories differ, so without a tier outside GPU memory the agents evict one another.

At eight agents the histories no longer fit in GPU memory. vLLM on its own recomputes them, and generation falls to 4.4 tokens per second per agent; with ADP a returning agent reads its history back off the accelerator and keeps 29.2. The median agent stays within 50 ms per token up to four concurrent agents without the connector, and all the way to 32 with it, the highest load we ran.

Most of the work in this release, and in the two before it, was the same kind of work: teaching the connector how another architecture lays out its cache. Mamba state, sliding windows, the indexers of sparse attention, MLA, MTP layers. Today 171 lines of the connector name a specific architecture, and every new model family has added to them.

vLLM has a second extension point that takes that work away: the KV offloading framework and its OffloadingSpec API. There the engine describes the cache itself. It reduces every layer to plain pages grouped by cache group, handles hybrid groups, Mamba alignment and MLA replication, and hands the backend nothing but pages and keys. A backend is registered from outside vLLM, with no fork.

We have started building the ADP backend on that API. In the prototype, the number of lines that name an architecture is zero. It passes its tests on vLLM 0.28 and 0.29 and runs end to end against the real gateway. Pages travel between GPU memory and the accelerator on the connector's own path rather than through vLLM's host-memory tier, which would add a copy. Validation on our GPU stands and tensor parallelism across multiple GPUs come next.

Instead of adapting the connector to each new model, ADP inherits vLLM's own model support: every model vLLM runs becomes a model ADP accelerates.

• Objects larger than the device accepts in one command are fragmented by the gateway, so KV pages of any size are written in full.
• A restored Mamba state lands in the block the engine uses next, including when the engine prepares that block early.
• The sparse-attention indexer reads the KV cache only after the restore has completed.
• The scheduler and the workers share one view of the layers and their ids; the connector exchanges exactly the caches vLLM hands over.
• I/O capacity is sized per submitted request, which keeps long prompts well within limits.
• After an EXIST timeout the client reopens its connection and carries on.
• The restore plan is recomputed when the local cache hit grows while a request waits, keeping Mamba, MLA and sliding-window models consistent.
• Data derived from an incomplete restore is never written back; the related requests and their writes are cleaned up.
• A restore error is attributed to the step where it occurs.
• Deletes wait for every outstanding read, and each MTP write is awaited exactly once.
• Early writes that include the MTP layer are fenced correctly.