超越事实：大型语言模型分布式阅读理解基准测试
Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models
AI总结：
本文提出Text2DistBench基准测试，评估LLM从自然语言中推断分布知识的能力，通过真实YouTube评论数据，测试模型在估计评论比例和识别高频主题方面的表现，揭示LLM在分布式阅读理解中的能力和局限。
摘要：
尽管大多数LLM阅读理解基准侧重于可以通过局部文本证据回答的事实性信息，许多现实任务需要理解分布性信息，如群体层面的趋势和偏好。我们引入Text2DistBench，一个用于评估LLM从自然语言中推断分布知识的阅读理解基准。该基准基于真实世界YouTube评论，提供实体元数据和相关评论，要求模型回答分布性问题，如估计正负评论比例或识别观众讨论最频繁和第二频繁的主题。为支持可靠和长期评估，Text2DistBench的构建流程完全自动化且持续更新以纳入新出现的实体。跨多个LLM的实验表明，尽管模型显著优于随机基线，但不同分布类型和特征的表现差异很大。这些发现突显了当前LLM在分布式阅读理解中的能力和局限，并展示了Text2DistBench作为未来研究实用且可扩展测试平台的价值。
英文摘要：
While most reading comprehension benchmarks for LLMs focus on factual information that can be answered by localizing specific textual evidence, many real-world tasks require understanding distributional information, such as population-level trends and preferences expressed across collections of text. We introduce Text2DistBench, a reading comprehension benchmark for evaluating LLMs' ability to infer distributional knowledge from natural language. Built from real-world YouTube comments about movie and music entities, the benchmark provides models with entity metadata and associated comments, and requires them to answer distributional questions, such as estimating the proportions of positive and negative comments, or identifying the most and second most frequent topics discussed among viewers. To support reliable and long-term evaluation, the construction pipeline of Text2DistBench is fully automated and continuously updated to incorporate newly emerging entities over time. Experiments across multiple LLMs show that while models substantially outperform random baselines, performance varies widely across different distribution types and characteristics. These findings highlight both the capabilities and limitations of current LLMs in distributional reading comprehension and demonstrate the value of Text2DistBench as a practical and scalable testbed for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
