5 different parallelism techniques:

Data Parallelism (DP): batch size dimensionTensor Parallelism (TP): hidden dimension (number of heads)Sequence and Context Parallelism (SP/CP): sequence length dimensionPipeline Parallelism (PP): model layersExpert Parallelism (EP): model experts/FNN in MoE

Along with DP, use these to reduce memory reduction:

ZeRO-1: sharding optimizer states among the DP replicasZeRO-2: sharding optimizer states and gradients among the DP replicasZeRO-3: sharding optimizer state...