KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

The era of giant LLM today calls forth distributed training. Despite countless distributed training frameworks that have been published in the past decade, few have excelled at real industry production, as the quality favored the most is often the Ease of Use instead of pure Performance. The Ease of Use lies in two essentials -- PyTorch and Automatic Parallelism, because: i) PyTorch ecosystem dominates and owns 92% of models on HuggingFace, and ii) giant models cannot be trained without complex nD Parallelism. Currently, this Ease of Use is "broken" for industry-level frameworks, as they are either not PyTorch-native (TensorFlow/JAX) or not fully Automated (Megatron/DeepSpeed/torch). We propose a novel framework that combines PyTorch Nativeness and Automatic Parallelism for scaling LLM training with Ease of Use. We only expect developers to write single-device torch code but automatically parallelize it into nD parallelism with all heavy lifting handled transparently.

当今巨型LLM时代呼唤分布式训练。尽管过去十年中已经发布了无数分布式训练框架，但很少有能够在真实产业生产中表现出色，因为最受青睐的质量往往是易用性而不是纯性能。易用性在于两个关键点--PyTorch和自动并行性，因为：i）PyTorch生态系统主导并拥有HuggingFace上92%的模型，ii）巨型模型无法在没有复杂的nD并行性的情况下进行训练。 目前，这种易用性对于产业级框架来说已经“破碎”，因为它们要么不是PyTorch原生的（TensorFlow/JAX），要么不是完全自动化的（Megatron/DeepSpeed/torch）。 我们提出了一个结合了PyTorch原生性和自动并行性的新型框架，以便通过易用性扩展LLM训练。我们只期望开发人员编写单设备torch代码，但自动将其并行化为nD并行性，所有繁重的工作都由框架透明地处理。