KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

Today, more and more car companies are building a new generation of Internet of Vehicles platforms based on cloud-native technology stacks such as Kubernetes. However, as more and more cars are produced, they generate hundreds of GB of data every second, making it difficult to store massive data in real-time and making storage costs difficult to control. which requires the platform's underlying database to be low-cost, high-performance, and efficient. openGemini is a cloud-native distributed time series database with high performance and low cost. In data writing, we provide a dedicated high-performance data writing component that supports Arrow Flight. Regarding data storage, we provide specialized data compression algorithms and support local data storage and object storage. This talk will introduce how to build Internet of Vehicles platforms based on cloud-native technology stacks and share the technical practices on how to efficiently manage massive vehicle data.

今天，越来越多的汽车公司正在基于Kubernetes等云原生技术堆栈构建新一代车联网平台。然而，随着汽车的生产越来越多，它们每秒产生数百GB的数据，使得实时存储海量数据变得困难，存储成本难以控制。这就要求平台的底层数据库要低成本、高性能和高效。openGemini是一个具有高性能和低成本的云原生分布式时间序列数据库。在数据写入方面，我们提供了支持Arrow Flight的专用高性能数据写入组件。在数据存储方面，我们提供了专门的数据压缩算法，并支持本地数据存储和对象存储。 本次演讲将介绍如何基于云原生技术堆栈构建车联网平台，并分享如何有效管理海量车辆数据的技术实践。

YDB is an open-source distributed database management system that, for performance reasons, uses raw disk drives (block devices) to store all data, without any filesystem. It was relatively straightforward to manage such setup in the bare-metal world of the past, but the dynamic nature of cloud-native environments introduced new challenges to keep this performance benefit. In this talk, we'll explore how to leverage Kubernetes and the Operator design pattern to modernize how stateful distributed database clusters are managed without changing the primary approach to how the data is physically stored.

YDB是一个开源的分布式数据库管理系统，为了性能考虑，使用原始磁盘驱动器（块设备）存储所有数据，而不使用任何文件系统。在过去的裸金属世界中管理这样的设置相对比较简单，但云原生环境的动态特性引入了新的挑战，以保持这种性能优势。在这次演讲中，我们将探讨如何利用Kubernetes和运算符设计模式来现代化管理有状态的分布式数据库集群，而不改变数据物理存储的主要方法。

Currently, most Windows CSI drivers depend on Windows csi-proxy because various privileged operations cannot be done from a containerized application running on a Windows node. Beginning in Kubernetes 1.23, HostProcess container is supported and it can run directly on the host as a regular process. Switching to HostProcess container deployment will make Windows CSI driver development and deployment easier. This session will cover the history and implementation details of Windows csi-proxy project, why csi-proxy is needed on Windows CSI driver starting in kubernetes 1.18, and why we removed this csi-proxy dependency from Kubernetes 1.26. We will explore the key learnings and gotchas we resolved while migrating Windows CSI driver development from csi-proxy dependent deployment to HostProcess container deployment. After attending this session, you will understand why and how to migrate your Windows applications to gain the benefits of using HostProcess containers.

目前，大多数Windows CSI驱动程序依赖于Windows csi-proxy，因为各种特权操作无法从在Windows节点上运行的容器化应用程序中执行。从Kubernetes 1.23开始，支持HostProcess容器，它可以直接在主机上作为常规进程运行。切换到HostProcess容器部署将使Windows CSI驱动程序的开发和部署变得更加简单。本场演讲将涵盖Windows csi-proxy项目的历史和实施细节，解释为什么从Kubernetes 1.18开始在Windows CSI驱动程序中需要csi-proxy，以及为什么我们在Kubernetes 1.26中删除了这种csi-proxy依赖性。我们将探讨在将Windows CSI驱动程序开发从依赖于csi-proxy的部署迁移到HostProcess容器部署时解决的关键问题和注意事项。参加本场演讲后，您将了解为什么以及如何将您的Windows应用程序迁移到使用HostProcess容器以获得更多好处。

Virtio-fs is a shared file system between virtual machines or secure containers and host, Storage Performance Development Kit(SPDK) vhost-fs is the backend implementation of virtio-fs in userspace, in this presentation, we will summarize typical storage solutions that use SPDK vhost-fs and components to build the storage stack, then go through the evolution of SPDK vhost-fs from BlobFS to latest FSDEV module, advanced features such as interrupt mode and thread modeling for data processing in SPDK vhost-fs are also covered.

Virtio-fs是虚拟机或安全容器与主机之间共享文件系统，Storage Performance Development Kit(SPDK) vhost-fs是virtio-fs在用户空间的后端实现。在本次演讲中，我们将总结使用SPDK vhost-fs和组件构建存储栈的典型存储解决方案，然后介绍SPDK vhost-fs从BlobFS到最新的FSDEV模块的演变过程，还将涵盖SPDK vhost-fs中用于数据处理的高级功能，如中断模式和线程建模。

The rapid advancement of artificial intelligence technologies, especially the development of large language models (LLMs), has led to a sharp increase in the amount of data that enterprises need to process. Managing large-scale data clusters in Kubernetes environments presents several challenges, including storage performance, complex access control management and system stability. JuiceFS is a distributed POSIX file system designed for cloud. It was open-sourced in 2021( 9.8k stars) To deliver an optimal experience in Kubernetes, JuiceFS developed JuiceFS CSI Driver. In addition, JuiceFS CSI introduced several new designs to support large-scale, complex AI training tasks such as the mount pod mode and the sidecar mode for serverless environments. Outline: - LLM Storage challenges - JuiceFS CSI Driver Architectural - Mount pod mode\Sidecar mode - Practical experience - Future

人工智能技术的快速发展，特别是大型语言模型（LLMs）的发展，导致企业需要处理的数据量急剧增加。在Kubernetes环境中管理大规模数据集群面临着多个挑战，包括存储性能、复杂的访问控制管理和系统稳定性。 JuiceFS是一种为云设计的分布式POSIX文件系统。它于2021年开源（拥有9.8k星）。为了在Kubernetes中提供最佳体验，JuiceFS开发了JuiceFS CSI驱动程序。此外，JuiceFS CSI引入了几项新设计，以支持大规模、复杂的人工智能训练任务，如挂载Pod模式和用于无服务器环境的Sidecar模式。 大纲： - LLM存储挑战 - JuiceFS CSI驱动程序架构 - 挂载Pod模式\Sidecar模式 - 实践经验 - 未来

In the past year, Kuaishou successfully migrated nearly 100,000 Redis instances from traditional bare metal environments to the Kubernetes platform, achieving a significant doubling of resource utilization. While ensuring business stability, this large-scale migration faced numerous challenges, including smooth migration execution, finding a balance between increasing deployment density (resource utilization) and ensuring system stability, avoiding interference with other services during coexistence, and addressing specific issues associated with stateful services like databases (including data management, configuration management, ensuring high availability, cross-cluster disaster recovery, etc.). This session will share Kuaishou's large-scale practical experience in Redis cloud-native transformation, in collaboration with the open-source project KubeBlocks, covering aspects such as smooth migration, resource efficiency improvement, and efficient database management.

在过去的一年中，快手成功将近10万个Redis实例从传统裸机环境迁移到Kubernetes平台，实现资源利用率显著翻倍。在确保业务稳定性的同时，这一大规模迁移面临诸多挑战，包括顺利执行迁移、在增加部署密度（资源利用率）和确保系统稳定性之间找到平衡、在共存期间避免与其他服务的干扰，以及解决与数据库等有状态服务相关的特定问题（包括数据管理、配置管理、确保高可用性、跨集群灾难恢复等）。 本场演讲将分享快手在Redis云原生转型方面的大规模实践经验，与开源项目KubeBlocks合作，涵盖顺利迁移、资源效率提升和高效数据库管理等方面。

The distributed and dynamic nature of Kubernetes makes data protection challenging to guarantee data availability and durability, below are summaries of the issues we encountered in the real customer environments: 1. Application definition and resources capture 2. Application data consistency 3.Application restore on heterogenous and across-cloud environments We provide the detailed description of these issues in the "Additional resources" section due to the character limitation of the "Description".

Kubernetes的分布式和动态特性使得数据保护变得具有挑战性，以确保数据的可用性和持久性。以下是我们在真实客户环境中遇到的问题摘要： 1. 应用程序定义和资源捕获 2. 应用程序数据一致性 3. 跨异构和跨云环境的应用程序恢复 由于“描述”部分的字符限制，我们将在“附加资源”部分提供这些问题的详细描述。

You might have heard of TiDB, a distributed open-source database known for its virtually limitless horizontal scalability, capable of handling both online transactional processing and analytical workloads while being compatible with the MySQL protocol. Traditionally, different databases have been employed to handle various workloads in our application architecture designs. Commonly, relational databases are used for online transaction processing, with data asynchronously distributed to analytical databases, document stores, and cache databases. With the rise of AI, an additional type of database needs consideration — the vector database. But introducing this type of database can add unnecessary complexity to your technology stack. This talk we will discuss how TiDB integrates multiple functionalities such as real-time transaction processing, online analytics, sharding-free architecture, and vector type computations, all aimed at reducing the cognitive load for developers.

您可能已经听说过 TiDB，这是一个分布式开源数据库，以其几乎无限的水平扩展性而闻名，能够处理在线事务处理和分析工作负载，同时兼容 MySQL 协议。 传统上，在我们的应用架构设计中，通常会使用不同的数据库来处理各种工作负载。通常情况下，关系数据库用于在线事务处理，数据会异步分布到分析数据库、文档存储和缓存数据库。随着人工智能的兴起，还需要考虑一种额外的数据库类型 —— 向量数据库。但引入这种类型的数据库可能会给您的技术堆栈增加不必要的复杂性。 在本次演讲中，我们将讨论 TiDB 如何集成多种功能，如实时事务处理、在线分析、无分片架构和向量类型计算，所有这些都旨在减少开发人员的认知负荷。