1. Cache size is architecture-dependent
The first thing to price is the cache itself. How many bytes must you reserve to store one sequence’s KV state?
For standard GQA and MHA, every token in every layer produces a key and a value vector:
$$M_{KV} = N_{seq} \cdot L \cdot N_{layers} \cdot (2 \cdot H_{KV} \cdot d_{head}) \cdot b$$
MLA compresses this by projecting to a low-rank latent, so the footprint drops to:
$$M_{MLA} = N_{seq} \cdot L \cdot N_{layers} \cdot (r_{KV} + d_{RoPE}) \cdot b$$
Newer recurrent or hybrid architectures can shrink the per-sequence state even further, sometimes to a fixed constant. The exact formula for each architecture is plugged into the calculator in the “Follow the calculation” section.
2. The value of a cache is bounded by the cost of recreating it
If you discard the KV state, the next request for the same prefix must re-run the prefill. That recomputation costs roughly:
$$t_{recompute} \approx \frac{2 \cdot N_{seq} \cdot L \cdot P_{active}}{F_{cluster}}$$
This is the ceiling on what caching is worth. No retention policy, offloading tier, or hit-rate assumption can make the cache more valuable than the work it eliminates. Every decision that follows is just a comparison against this ceiling.
3. Cache duration has a storage cost
Memory is not free; it is rented by the second. If you keep the cache alive for duration $T$, you pay:
$$C_{rent}(T) = M_{cache} \cdot r_{tier} \cdot T$$
This is why the TTL chart is immediately legible: recomputation is a fixed lump-sum cost for a given workload, while storage cost rises linearly with time. The longer you hold the cache, the more you must believe a hit will arrive to justify the rent.
4. Offloading trades cheaper rent for slower retrieval
You can move the cache to a cheaper tier—DDR, SSD, or even HDD. The rent drops, but retrieval now incurs a transfer penalty:
$$t_{restore} = \frac{M_{cache}}{BW_{tier}}$$
The expected cost of a cached strategy therefore blends the rent, the retrieval cost weighted by hit rate, and the recomputation cost weighted by miss rate:
$$\mathbb{E}[C_{cache}] = C_{rent} + h \cdot C_{restore} + (1 - h) \cdot C_{recompute}$$
DDR, SSD, and HDD are simply points along a spectrum: lower rent, higher retrieval latency. The calculator lets you slide along that spectrum and see where the optimum lands.
5. The break-even TTL falls directly out of the inequality
Start with the decision rule: caching beats recomputation when its expected cost is lower.
$$C_{rent} + h \cdot C_{restore} + (1 - h) \cdot C_{recompute} < C_{recompute}$$
Rearranging:
$$C_{rent} < h \cdot (C_{recompute} - C_{restore})$$
Substitute $C_{rent} = M_{cache} \cdot r_{tier} \cdot T$ and solve for $T$:
$$T^{*} = \frac{h \cdot (C_{recompute} - C_{restore})}{M_{cache} \cdot r_{tier}}$$
This is the core predictive lens. It tells you, in closed form, why five forces push you toward recomputation:
- Longer TTL → more rent accumulated.
- Larger cache → more bytes to rent.
- Expensive tier → higher rent per byte.
- Low hit rate → the left side is weighted by a small $h$.
- Slow retrieval → $C_{restore}$ approaches $C_{recompute}$, shrinking the numerator.
The general decision rule is simply the minimum across all available strategies:
$$s^{*} = \arg\min_{s \in \{\text{HBM}, \text{DDR}, \text{SSD}, \text{HDD}, \text{recompute}\}} \mathbb{E}[C_s]$$