[CHEATSHEET] Optimization that makes server performance realistic.

Problem: Long prompts (L=32k to 128k) exceed GPU memory if processed in one shot. Additionally, a long prompt will keep other shorter prompts waiting as server will be busy.

Solution: Process the prompt in chunks of size L_chunk sequentially, while building the same KV cache.

Full input: [B, L, d_model]Split into: [B, L_chunk, d_model] × N_chunksKV cache after processing all chunks: [B, n_heads, L, d_head]

Masking

At chunk i, self-att...