Each request carries the cached prefix plus new tool output. While the request runs, its KV blocks are held on the GPU. When the agent goes back to waiting on a tool, the GPU lets go of them: they go on the belt and move toward the edge, pushed by every other request. Will they still be there when the agent comes back?
First request: system prompt, skills and user task.
Each turn: the agent sends the prefix plus new tool output, the GPU looks up the prefix blocks on the belt and pulls them back in (a cache hit), adds a slot for the new tool output, and returns a tool call. While the agent waits, every block goes back on the belt. When the GPU pulls last turn’s orange block back in, it turns blue: it is now part of the prefix. Blocks that fell off the edge are recomputed.