Memory components in LLM
LLMs are memory hungry. GPUs are limited by memory more compared to compute.
Forward: [Parameters] -> [Activations]
Backward: [Activations] -> [Gradients]
[Gradients] + [Optimizer States] -> [Updated Parameters]
Memory usage across sequence length in different LLM:
(“dotted line” is the memory available in 1 GPU; Note 8B parameter models do not fit in one GPU.)
Note:
Parameters, gradients and optimizer states remain constant.Only, activation is dependent on sequence length.In inference, only parameter ...
No comments have been added yet.
