arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.03437cs.LGcs.CLstat.ML
收藏

基于m依赖性的文本分段一致性核变化点检测

Consistent Kernel Change-Point Detection under m-Dependence for Text Segmentation

Jairo Diaz-Rodriguez, Mumin Jia

AI总结:

本文提出在m依赖数据下基于核变化点检测的文本分段方法,通过理论证明和实验证明其在文本分段任务中的有效性。

摘要:

核变化点检测(KCPD)已成为识别复杂数据中结构变化的广泛工具。尽管现有理论在独立性假设下建立了一致性,但现实世界中的序列数据如文本表现出强烈的依赖性。我们为KCPD在m依赖数据下建立了新的保证:具体而言,在温和的额外假设下,我们证明了检测到的变化点数量的一致性和位置的弱一致性。我们进行了一种基于大语言模型的模拟,生成合成的m依赖文本以验证渐近性。为了补充这些结果,我们展示了KCPD在文本分段中的首次全面实证研究,使用现代嵌入。在多样化的文本数据集中,KCPD与文本嵌入相比,在标准文本分段指标上优于基线。通过在泰勒·斯威夫特的推文中进行案例研究,我们证明KCPD不仅提供了强大的理论和模拟可靠性,而且在文本分段任务中具有实际有效性。

英文摘要:

Kernel change-point detection (KCPD) has become a widely used tool for identifying structural changes in complex data. While existing theory establishes consistency under independence assumptions, real-world sequential data such as text exhibits strong dependencies. We establish new guarantees for KCPD under $m$-dependent data: specifically, we prove consistency in the number of detected change points and weak consistency in their locations under mild additional assumptions. We perform an LLM-based simulation that generates synthetic $m$-dependent text to validate the asymptotics. To complement these results, we present the first comprehensive empirical study of KCPD for text segmentation with modern embeddings. Across diverse text datasets, KCPD with text embeddings outperforms baselines in standard text segmentation metrics. We demonstrate through a case study on Taylor Swift's tweets that KCPD not only provides strong theoretical and simulated reliability but also practical effectiveness for text segmentation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑