4 min read
MegaTrain fits 120B LLM training on one…
MegaTrain stores weights in host memory and streams one layer at a time to the GPU, training 120B models on a single H200 with 1.5TB RAM. It beats DeepSpeed ZeRO-3 offload by 1.84x at 14B scale.
MegaTrain stores weights in host memory and streams one layer at a time to the GPU, training 120B models on a single H200 with 1.5TB RAM. It beats DeepSpeed ZeRO-3 offload by 1.84x at 14B scale.