How GPUs Talk: A Practical Guide to Multi-GPU Training and Communication
Data parallel, FSDP, tensor parallel, pipeline parallel, and the collectives that hold them together.
Data parallel, FSDP, tensor parallel, pipeline parallel, and the collectives that hold them together.
Want to get notified when new posts go live?
✉️ Email me to subscribe