LiveLongBench:应对直播口语文本的长上下文理解
LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams
AI总结:
针对直播口语文本的长上下文理解,构建首个冗余丰富的口语长文本基准LiveLongBench,评估现有方法并提出新基线,填补评估空白,助力电商应用。
摘要:
长上下文理解在自然语言处理中构成重大挑战,尤其是在以基于语音的元素、高冗余度和不均匀信息密度为特征的真实世界对话中。尽管大型语言模型(LLMs)在现有基准上取得了令人印象深刻的结果,但这些数据集未能反映此类文本的复杂性,限制了其在实际场景中的适用性。为弥合这一差距,我们构建了首个源自直播的口语长文本数据集,旨在反映真实世界场景中冗余丰富和对话性的特点。我们构建了三类任务:检索依赖型、推理依赖型和混合型。随后,我们评估了流行的LLMs和专门方法,以评估它们在这些任务中理解长上下文的能力。我们的结果表明,现有方法表现出强烈的任务特定偏好,在高冗余输入上表现不佳,且没有单一方法能始终优于其他方法。我们提出了一种新的基线,能更好地处理口语文本中的冗余,并在各任务中取得强劲表现。我们的发现凸显了现有方法的关键局限性,并为改进长上下文理解提出了未来方向。最后,我们的基准填补了评估长上下文口语理解的空白,并为开发真实世界电子商务系统提供了实用基础。代码和基准可在https://github.com/Yarayx/livelongbench获取。
英文摘要:
Long-context understanding poses significant challenges in natural language processing, particularly for real-world dialogues characterized by speech-based elements, high redundancy, and uneven information density. Although large language models (LLMs) achieve impressive results on existing benchmarks, these datasets fail to reflect the complexities of such texts, limiting their applicability to practical scenarios. To bridge this gap, we construct the first spoken long-text dataset, derived from live streams, designed to reflect the redundancy-rich and conversational nature of real-world scenarios. We construct tasks in three categories: retrieval-dependent, reasoning-dependent, and hybrid. We then evaluate both popular LLMs and specialized methods to assess their ability to understand long-contexts in these tasks. Our results show that current methods exhibit strong task-specific preferences and perform poorly on highly redundant inputs, with no single method consistently outperforming others. We propose a new baseline that better handles redundancy in spoken text and achieves strong performance across tasks. Our findings highlight key limitations of current methods and suggest future directions for improving long-context understanding. Finally, our benchmark fills a gap in evaluating long-context spoken language understanding and provides a practical foundation for developing real-world e-commerce systems. The code and benchmark are available at https://github.com/Yarayx/livelongbench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。