KubeCon + CloudNativeCon + Open Source Summit + AI_dev China 2024

Debugging Kubernetes system issues is complicated: different controllers manipulate objects independently, sometimes triggering changes in other controllers. Unlike traditional RPC-based services, the relationship between components is not explicit; identifying which component causes an issue could be like finding a needle in a haystack. Components expose their own fragmented data, often limited to the lifecycle of a single request and fail to illustrate the bigger picture of asynchronous causal events. This talk introduces Kelemetry, a global tracing system for the Kubernetes control plane using scattered data sources from audit log, events, informers and component traces. Through several demonstrations of troubleshooting online problems, we will see how Kelemetry reveals the state transition of related objects over a long timespan and reconstructs the causal hierarchy of events to provide intuitive insight into the What, When and Why of everything going on in a Kubernetes system.

调试Kubernetes系统问题是复杂的：不同的控制器独立地操作对象，有时会触发其他控制器的变化。与传统的基于RPC的服务不同，组件之间的关系并不明确；确定哪个组件引起了问题就像在一堆草堆中找针一样困难。组件展示它们自己的碎片化数据，通常仅限于单个请求的生命周期，并未展示异步因果事件的整体情况。 本次演讲介绍了Kelemetry，这是一个利用审计日志、事件、通知器和组件跟踪的分散数据源的Kubernetes控制平面全局跟踪系统。通过几次在线问题排查演示，我们将看到Kelemetry如何揭示相关对象在长时间跨度内的状态转换，并重建事件的因果层次结构，以提供对Kubernetes系统中发生的一切的直观洞察。