arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.17399cs.LGcs.AIstat.ML
收藏

通过投注式监测学习检测非结构化数据流中的分布偏移

Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting

University of California, San Francisco(加利福尼亚大学旧金山分校);University of Maryland(马里兰大学);Johns Hopkins University(约翰斯·霍普金斯大学)

Siyu Zhou, Jialin Ouyang, Fan Xia, Adarsh Subbaswamy, Drew Prinster, Suchi Saria, Augusto Garcia-Agundez, Sarah Pollet, Jinoos Yazdany, Patrick Vossler, Jean Feng

AI总结:

该研究提出投注式监测方法,开发渐近e-检测器及自适应监测策略,在三个真实世界数据集上实现非结构化数据流分布偏移的快速、可解释检测,优于固定监测器。

摘要:

生成式AI系统的持续监测需要检测其摄入和生成的非结构化数据中的分布偏移。现有监测方法通常通过要求用于检测偏移的差异度量为预先指定来控制误报。然而,对于非结构化数据,预先指定会产生灵敏度-广度权衡:狭义定义的度量对预期偏移敏感,但可能遗漏意外变化;而广义定义的度量可检测更广泛的偏移,但灵敏度降低。尽管自适应假设偏移方向可克服此限制,但这种依赖数据的自适应通常会使传统的误报保证失效。我们通过投注式测试协调了自适应与统计有效性,将序贯变化点检测重新表述为一个投注游戏,其中投注者随着新观测的到来不断更新其假设的偏移方向或见证函数。我们引入渐近e-检测器,其使用大小为m的校准集近似变化前分布,并证明随着m增大,它们渐近控制误报的平均运行长度。经验表明,小至m=30的校准集即可提供有效控制。随后,我们开发了自适应监测策略,范围从从嵌入中学习的黑箱投注者到基于LLM提出概念的可解释投注者。在三个真实世界数据集(包括临床AI记录部署的数据)上,自适应监测器在相同误报水平下比固定监测器更快检测到偏移。此外,当真实偏移稀疏且可解释时,LLM概念监测器提供最快且最可解释的检测,同时还能为复杂偏移指导根因诊断。

英文摘要:

Continuous monitoring of generative AI systems requires detecting distribution shifts in the unstructured data they ingest and generate. Existing monitoring methods typically control false alarms by requiring the discrepancy measure used to detect shifts to be *pre-specified*. However, for unstructured data, pre-specification creates a sensitivity--breadth trade-off: a narrowly defined measure is sensitive to the anticipated shift but may miss unexpected changes, whereas a broadly defined measure may detect a wider range of shifts but with reduced sensitivity. Although adaptively hypothesizing the shift direction can overcome this limitation, such data-dependent adaptation generally invalidates conventional false-alarm guarantees. We reconcile adaptivity with statistical validity through testing-by-betting, reframing sequential changepoint detection as a betting game in which the bettor continually updates its hypothesized shift direction, or witness function, as new observations arrive. We introduce *asymptotic e-detectors*, which approximate the pre-change distribution using a calibration set of size $m$, and prove that they asymptotically control the average run length to a false alarm as $m$ increases. Empirically, calibration sets as small as $m=30$ provide effective control. We then develop adaptive monitoring strategies ranging from black-box bettors that learn from embeddings to interpretable bettors based on LLM-proposed concepts. Across three real-world datasets, including data from a clinical AI scribe deployment, adaptive monitors detect shifts faster than fixed monitors at the same false-alarm level. Further, when the true shifts are sparse and interpretable, LLM-concept monitors provide the fastest and most interpretable detection, while also guiding root-cause diagnosis for complex shifts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑