When you have 100K GPUs, you want to utilize all of them.

Given a DL model (like LLM) and thousands of GPUs, how will you distribute your workload efficiently?

Parallelism rules:

Start with Data Parallelism (DP) = simplest, scales until gradient communication dominates.Add Tensor Parallelism (TP) = when layers (matmuls, attention) are too big for a single GPU.Add Pipeline Parallelism (PP) = when model depth is too large for memory of one GPU.Add Expert Parallelism (EP) = when compute cost grows, us...