KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

With the popularity of LLM, large-scale pre-training has become an indispensable step in AI research and implementation. However, large-scale distributed parallel training requires developers to consider various factors affecting the efficiency of model development and training, such as partitioning and communication, and then modify the model accordingly. In this presentation, we will demonstrate an automatic parallelization approach that allows developers to focus on algorithm research without the need for intrusive model modifications. Distributed training on a large-scale cluster can be achieved simply by configuring strategies. Developers can also utilize MindSpore's hyperparameter search model to automatically find the best parallelization strategy. The parallel strategy obtained through search can achieve 90%-110% of the expert tuning performance, significantly reducing the time required for model modifications while efficiently accelerating LLM training.

随着LLM的流行，大规模预训练已成为人工智能研究和实施中不可或缺的一步。然而，大规模分布式并行训练需要开发人员考虑各种影响模型开发和训练效率的因素，如分区和通信，然后相应地修改模型。 在本次演示中，我们将展示一种自动并行化方法，使开发人员能够专注于算法研究，而无需进行侵入性的模型修改。通过配置策略，可以简单实现在大规模集群上的分布式训练。开发人员还可以利用MindSpore的超参数搜索模型自动找到最佳的并行化策略。通过搜索获得的并行策略可以实现专家调整性能的90%-110%，显著减少了模型修改所需的时间，同时有效加速LLM的训练。