KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

LLMs have heightened public expectations of generative models. However, as noted in the Gartner report, running AI applications in production poses significant challenges. To tackle the challenges, we have redesigned and optimized the software capabilities of Cloud Native AI Technologies. By extending KServe to handle OpenAI's streaming requests, it can accommodate the inference load of LLM. With Fluid and Vineyard, It shows a result of reducing Llama-30B model loading time from 10 minutes to under 25 seconds. However, the above optimizations do not stop there. Since LLM loading is not a high-frequency operation,It is crucial to utilize cronHPA for timed auto-scaling in order to achieve a balance between cost and performance, and to evaluate the cost-effectiveness of the scaling process. As KServe and Fluid's reviewer and maintainer, we share our insights on the challenges in the session. We will showcase effective use of Cloud Native AI and share our experiences in production.

LLM让公众对生成式大模型的期望提高。然而，正如Gartner报告所指出的，将AI应用程序投入生产中存在重大挑战。为了解决这些挑战，我们重新设计和优化了云原生AI技术的软件能力。通过扩展KServe以处理OpenAI的流式请求，它可以容纳LLM的推理负载。通过Fluid和Vineyard，我们成功将Llama-30B模型的加载时间从10分钟缩短到不到25秒。然而，上述优化并不止于此。由于LLM加载不是高频操作，利用cronHPA进行定时自动扩展至关重要，以实现成本和性能之间的平衡，并评估扩展过程的成本效益。作为KServe和Fluid的审阅者和维护者，我们在本场演讲中分享了对挑战的见解。我们将展示云原生AI的有效使用，并分享我们在生产中的经验。