KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

As LLMs continue to grow in size, their deployment and delivery in cloud-edge environments are faced with substantial challenges, especially within edge computing settings that encompass multiple sites with thousands of edge nodes. In this presentation, we will explore how to efficiently distribute LLM applications across dispersed edge nodes using OpenYurt. We will also delve into how Dragonfly’s P2P image distribution technology can address the issue of public network bandwidth consumption encountered during cross-site transmission, reducing public network traffic consumption by up to 90% compared to conventional LLM distribution, and achieving rapid and efficient sharing of LLMs in physically isolated environments. During this presentation, container service experts from Alibaba Cloud and Ant Group will share this solution and introduce the practical application of combining OpenYurt with Dragonfly in edge computing scenarios for LLMs.

随着LLM的规模不断增长，它们在云边缘环境中的部署和交付面临着重大挑战，特别是在涵盖数千个边缘节点的边缘计算环境中。在本次演讲中，我们将探讨如何使用OpenYurt在分散的边缘节点上高效分发LLM应用程序。我们还将深入探讨Dragonfly的P2P图像分发技术如何解决跨站点传输中遇到的公共网络带宽消耗问题，与传统的LLM分发相比，将公共网络流量消耗降低高达90％，实现在物理隔离环境中LLM的快速高效共享。 在本次演示中，来自阿里巴巴云和蚂蚁集团的容器服务专家将分享这一解决方案，并介绍在LLM的边缘计算场景中将OpenYurt与Dragonfly结合应用的实际应用。