I'll rewrite this later to thread the relevant parts throughout the piece instead of explaining the full analogy up front. But just keeping here for reference.

Here’s a simple analogy: imagine you’re reading a book to write a report, taking notes on scratch paper as you go. Ordinarily you’d throw that scratch paper away after a session, but if you keep it on your desk, you can consult your notes instead of re-reading the whole thing. That’s what we call a “cache hit.” The tradeoff is that your notes consume scarce desk space. You can file those notes in a cabinet instead, which costs cheaper space but makes retrieval slower. And keeping the notes only pays off if you (or a colleague) will ask for that book again.

In a transformer, the "notes" you take while reading are the Key-Value (KV) states computed during the forward pass. Inference must generate these states anyway — ie. the model needs “scratch paper” to generate output tokens. But taking those notes is not the same as storing them for later reuse, and that’s where caching comes into play. Now you consider retention policies (how long the paper stays on the desk), indexing (a label so you can find it), routing (which cabinet to check), among other considerations.

1. Cache size is architecture-dependent

For standard GQA/MHA:

M_KV = N_seq L N_layers (2 H_KV d_head) b

For MLA:

M_MLA = N_seq L N_layers (r_KV + d_RoPE) b

And newer recurrent/hybrid architectures can add fixed per-sequence state. The formula for each architecture is specified in the "Follow the calculation" section

2. Caching saves compute costs by spending memory

If you throw the prefix away, you eventually have to prefill it again:

t_recompute ≈ (2 N_seq L P_active) / F_cluster

The key idea: the economic value of cached state is bounded by the cost of recreating it.

3. Cache duration has a storage cost

C_rent(T) = M_cache · r_tier · T

This is the thing I most want explained because it makes the TTL chart immediately legible: recomputation is basically fixed for a given workload; storage costs generally rise linearly with cache duration.

4. Offloading to disk trades cheaper capacity for slower retrieval

t_restore = M_cache / BW_tier

and

E[C_cache] = C_rent + h C_restore + (1 − h) C_recompute

DDR/SSD/HDD gives you lower storage cost while consuming more serving time when the cache is actually reused.

5. The break-even TTL falls directly out of the inequality

Start with:

C_rent + h C_restore + (1 − h) C_recompute < C_recompute

Rearrange:

C_rent < h(C_recompute − C_restore)

And because (C_rent = M r T):

T* = h(C_recompute − C_restore) / (M_cache r_tier)

This is the core predictive lens of the explainer. It tells you directly why longer TTLs, larger caches, expensive storage, low hit rates, and slow retrieval each push you toward recomputation.