[CHEATSHEET] This works best with continuous batching + chunk prefill.

Problem: Decode is sequential and under-utilizes GPUs (

Solution: Always maximize the batch size of active decode requests per step, so the GPU runs one large kernel instead of many tiny ones.

At decode step t:

Suppose active requests = 50.Instead of launching 50 tiny decodes:Inputs merged → [B=50, 1, d_model]Outputs → [B=50, 1, V]

Each request still attends its own KV cache:

KV_i (KV cache of ith decode): [1, n_heads, (L_i + t), d...