Aditya Chatterjee's Blog

Not long ago, the notion of launching a web application without writing any code seemed unrealistic. Today it has become a practical reality for founders, marketers, and small teams who, facing tight deadlines, need to ship their products quickly and without technical delays. No-code AI platforms pair visual editors with machine learning. The real question is no longer whether these tools can build an app, but whether that app can grow. Scalability is what divides a weekend prototype from a prod...

5 different parallelism techniques:

Data Parallelism (DP): batch size dimensionTensor Parallelism (TP): hidden dimension (number of heads)Sequence and Context Parallelism (SP/CP): sequence length dimensionPipeline Parallelism (PP): model layersExpert Parallelism (EP): model experts/FNN in MoE

Along with DP, use these to reduce memory reduction:

ZeRO-1: sharding optimizer states among the DP replicasZeRO-2: sharding optimizer states and gradients among the DP replicasZeRO-3: sharding optimizer state...

LLMs are memory hungry. GPUs are limited by memory more compared to compute.

Forward: [Parameters] -> [Activations]

Backward: [Activations] -> [Gradients]

[Gradients] + [Optimizer States] -> [Updated Parameters]

Memory usage across sequence length in different LLM:

(“dotted line” is the memory available in 1 GPU; Note 8B parameter models do not fit in one GPU.)

Note:

Parameters, gradients and optimizer states remain constant.Only, activation is dependent on sequence length.In inference, only parameter ...

[CHEATSHEET] Separate GPUs for Prefill and Decode.

Note:

In continuous batching, if prefill and decode and merged, then time is wasted for decode.Prefill GPU occupancy ~ 90%Decode GPU occupancy ~30% and limited by memory bandwidth.Prefill prefers TP and DP parallelism.Decode prefers PP.

Core idea of P/D disaggregation:

Schedule prefill and decode requests in separate GPUs.2P1D = 2 GPUs for prefill requests. 1 GPU for decode requests.Prefill requests can be batched together.Decode requests can be bat...

[CHEATSHEET] Use multiple GPUs and pipeline.

Idea: Merge Decode and Prefill phase of different requests (in batch dimension) and operate on this batched input. Use Paged KV cache.

This is Continuous Batching with 4 requests (R1, R2, R3, R4) all executing on 1 GPU.

The smaller uniform blocks are decode phase.Larger 3 phases are prefill phase.

At every instance, decode and prefill phases from different requests are merged in batch (B) dimension. So, one operation is executing at a time. This one opera...

[CHEATSHEET] This works best with continuous batching + chunk prefill.

Problem: Decode is sequential and under-utilizes GPUs (

Solution: Always maximize the batch size of active decode requests per step, so the GPU runs one large kernel instead of many tiny ones.

At decode step t:

Suppose active requests = 50.Instead of launching 50 tiny decodes:Inputs merged → [B=50, 1, d_model]Outputs → [B=50, 1, V]

Each request still attends its own KV cache:

KV_i (KV cache of ith decode): [1, n_heads, (L_i + t), d...

[CHEATSHEET] Optimization that makes server performance realistic.

Problem: Long prompts (L=32k to 128k) exceed GPU memory if processed in one shot. Additionally, a long prompt will keep other shorter prompts waiting as server will be busy.

Solution: Process the prompt in chunks of size L_chunk sequentially, while building the same KV cache.

Full input: [B, L, d_model]Split into: [B, L_chunk, d_model] × N_chunksKV cache after processing all chunks: [B, n_heads, L, d_head]

Masking

At chunk i, self-att...

[CHEATSHEET] Understand the overhead of LLM Inference.

Split across prefill and decode:

Prefill:~60% MLP~35% Attention15% QKV projection10% Attention score & softmax10% Output projection)~5% Layernorm + OtherDecode:~60–70% Attention~25–30% MLP~5% Other

Prefill phase (compute-bound)

FFN dominates wall-clock time (since FLOPs ≈ proportional to L).

Decode phase (memory-bound)

Attention (KV reads/writes) dominates.

Why Different in Decode?

In decode, each new token attends over all past tokens (L+t).KV cach...

Inference = Prefill + Decode

LLM Inference is split in 2 components:

PrefillDecodes

Prefill (Prompt Encoding)

Input prompt tokens = processed in parallel through entire Transformer stack.Characteristics:Compute-bound (large MatMuls, parallelizable).KV cache is built (for decode).

Decode (Autoregressive Generation)

Model generates one token at a time, conditioned on KV cache + past tokens. Characteristics:

Memory-bound (repeated KVcache lookups, smaller MatMul).Sequential dependency = less parallelism.D...

In safety-critical, long-lifecycle systems = obsolescence is the real bottleneck. Adaptive architectures (FPGAs, SoCs, stable CPUs) = keep innovation alive after deployment.

GPU has this disadvantage.

Metric:

Most benchmarks = Performance per Watt (instant efficiency).Critical domains = Performance per Lifecycle (20+ yrs).

Why Lifecycle Performance Matters?

Long Standards Cycles = 20+ yrsHigh Risk = Recall cost millions, downtime catastrophic, cannot refresh hardware but innovation must continuePlatf...