后见之明的好处:追踪分布式系统中的边缘情况
The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems
摘要:
今天的分布式跟踪框架不足以对罕见的边缘情况请求进行故障排除。问题的关键在于专用性和开销之间的权衡。一方面,框架可以在请求进入系统时不加区别地选择要跟踪的请求(头部采样),但这不太可能捕获相关的边缘情况跟踪,因为框架直到事后才知道哪些请求会有问题。另一方面,框架可以跟踪所有内容,然后只保留有趣的边缘情况跟踪(尾部采样),但这对被跟踪的应用程序和巨大的数据摄取成本有很高的开销。在本文中,我们规避了这种权衡的任何边缘情况下,可以编程检测到的症状,如长尾延迟,错误,并检查队列。我们提出了一个轻量级的和永远在线的分布式跟踪系统,后见之明,它实现了一个追溯采样抽象:而不是急切地ingredient和处理的痕迹,后见之明懒洋洋地检索跟踪数据,只有在检测到问题的症状。后见之明类似于汽车仪表盘摄像头,在检测到动力突然震动时,会保留最后一个小时的镜头。使用后见之明的开发人员可以获得他们想要的精确的边缘情况跟踪,而不会产生不必要的开销或依赖运气。我们的评估表明,后见之明可扩展到每秒数百万个请求,增加纳秒级的开销来生成跟踪数据,每个节点处理GB/s的数据,透明地与现有的分布式跟踪系统集成,并在检测到边缘问题时,成功地在现实世界的用例中保持完整,详细的跟踪。
英文摘要:
Today's distributed tracing frameworks are ill-equipped to troubleshoot rare edge-case requests. The crux of the problem is a trade-off between specificity and overhead. On the one hand, frameworks can indiscriminately select requests to trace when they enter the system (head sampling), but this is unlikely to capture a relevant edge-case trace because the framework cannot know which requests will be problematic until after-the-fact. On the other hand, frameworks can trace everything and later keep only the interesting edge-case traces (tail sampling), but this has high overheads on the traced application and enormous data ingestion costs. In this paper we circumvent this trade-off for any edge-case with symptoms that can be programmatically detected, such as high tail latency, errors, and bottlenecked queues. We propose a lightweight and always-on distributed tracing system, Hindsight, which implements a retroactive sampling abstraction: instead of eagerly ingesting and processing traces, Hindsight lazily retrieves trace data only after symptoms of a problem are detected. Hindsight is analogous to a car dash-cam that, upon detecting a sudden jolt in momentum, persists the last hour of footage. Developers using Hindsight receive the exact edge-case traces they desire without undue overhead or dependence on luck. Our evaluation shows that Hindsight scales to millions of requests per second, adds nanosecond-level overhead to generate trace data, handles GB/s of data per node, transparently integrates with existing distributed tracing systems, and successfully persists full, detailed traces in real-world use cases when edge-case problems are detected.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。