KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

In the LLM training and inference era, the bottle neck has changed from computing to network. A lot of high throughput and low latency inter-connect technology are widely used, e.g. nvlink, nvswitch to build hyper computer such as nvidia super pod, google multi-slice, AWS placement group. However, Kubernetes has net yet addressed topology awareness efficiently, resulting in low performance when sub-optimal resources are provisioned. This talk will explore the inter-node communication and resources within node inter-connect. Also analyze how these two toplogical factors impacts on the runtime performance of AI workload especially for large language model training. The talk will cover: - How to model the topology on underlying resources like NUMA, Rack, Super Pod, Hyper Computer - How to make scheduler to aware of topology and make the best scheduling - How to coordinate topology-aware scheduling with DRA on node

在LLM训练和推断时代，瓶颈已经从计算转变为网络。许多高吞吐量和低延迟的互连技术被广泛使用，例如nvlink、nvswitch用于构建超级计算机，如nvidia超级Pod、谷歌多片、AWS放置组。 然而，Kubernetes尚未有效地解决拓扑意识问题，导致在资源配置不佳时性能较低。 本次演讲将探讨节点间通信和节点内部资源的互连。还将分析这两个拓扑因素如何影响AI工作负载的运行性能，特别是对于大型语言模型训练。 演讲内容包括： - 如何对底层资源（如NUMA、机架、超级计算机）建模拓扑 - 如何使调度程序意识到拓扑并进行最佳调度 - 如何协调拓扑感知调度与节点上的DRA