KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

The fault tolerance during train, fine-tuning, and even inferencing is crucial to modern AI workloads when it happens on large scale, with loads of GPU clusters. For training and fine-tuning tasks, failure of GPUs, storages, any hardware issues often cause the extending the training time to weeks and even months significantly. For inferencing, when massive loads of requests income, if one of the inferencing servers went faulty, we need a policy and scheduler to perform mitigation to transfer the workloads fast and efficiently. In this talk, We will introduce a series of mechanism we have designed to help Kubernetes clusters and workloads itself to locate, diagnostic the root cause, schedule and perform mitigation when it comes to any of hardware or CUDA API call failures to reduce the overall operating challenges. But the possibilities will not stop here, the fault awareness and mitigation scheduler will help any of the workloads to mitigate during failures.

在大规模GPU集群上进行训练、微调甚至推理时的容错性对现代人工智能工作负载至关重要。 对于训练和微调任务，GPU、存储等硬件故障经常会导致训练时间延长至数周甚至数月。对于推理任务，当大量请求涌入时，如果其中一个推理服务器出现故障，我们需要一种策略和调度程序来快速高效地转移工作负载。 在本次演讲中，我们将介绍一系列我们设计的机制，帮助Kubernetes集群和工作负载本身定位、诊断根本原因，并在硬件或CUDA API调用失败时进行调度和执行缓解，以减少整体运营挑战。但可能性不会止步于此，故障感知和缓解调度程序将帮助任何工作负载在故障期间进行缓解。