评估LLM在近期开放领域问题上的可靠性
Assessing LLM Reliability on Temporally Recent Open-Domain Questions
AI总结：
RECOM研究通过分析15000个Reddit问题，发现LLM在语义对齐上表现优异，但词汇重合度低，揭示模型通过改写保留意义，挑战传统词汇度量的可靠性。
摘要：
大型语言模型（LLMs）越来越多地被用于开放领域问答，但它们与人类在时间最近信息上的对齐仍缺乏深入研究。我们介绍了RECOM（Reddit评估模型对应性基准数据集），该数据集包含15000个来自2025年9月的Reddit问题，配以社区生成的参考答案。我们研究了四个开源LLM（Llama3.1-8B、Mistral-7B、Gemma-2-9B和GPT-OSS-20B）如何回应这些问题，评估对齐情况使用词汇度量（BLEU、ROUGE）、语义相似性（BERTScore、MoverScore、余弦相似度）和逻辑推理（NLI）。我们的核心发现是一个显著的语义-词汇悖论：所有模型在参考上达到超过99%的余弦相似度，尽管BLEU-1重叠率低于8%，差距达90多个百分点，表明模型通过广泛改写而非词汇复现来保留意义。MoverScore（51-53%）证实了这一模式，处于中间位置，反映了语义对齐的最佳传输成本。此外，模型规模不预测性能：Mistral-7B（7B参数）在所有指标上均优于GPT-OSS-20B（20B参数）。NLI分析显示，矛盾率仍低于7%，表明模型很少生成直接与人类共识冲突的内容。这些发现挑战了用词汇度量评估抽象生成的可靠性，并呼吁多维评估框架，以捕捉超越表面文本匹配的语义忠实性。RECOM数据集在https://anonymous.4open.science/r/recom-D4B0公开。
英文摘要：
Large Language Models (LLMs) are increasingly deployed for open-domain question answering, yet their alignment with human perspectives on temporally recent information remains underexplored. We introduce RECOM (Reddit Evaluation for Correspondence of Models), a benchmark dataset of 15,000 recent Reddit questions from September 2025 paired with community-derived reference answers. We investigate how four open-source LLMs (Llama3.1-8B, Mistral-7B, Gemma-2-9B, and GPT-OSS-20B) respond to these questions, evaluating alignment using lexical metrics (BLEU, ROUGE), semantic similarity (BERTScore, MoverScore, cosine similarity), and logical inference (NLI). Our central finding is a striking semantic-lexical paradox: all models achieve over 99% cosine similarity with references despite less than 8% BLEU-1 overlap, a 90+ percentage point gap indicating that models preserve meaning through extensive paraphrasing rather than lexical reproduction. MoverScore (51-53%) confirms this pattern, occupying an intermediate position that reflects the optimal transport cost of semantic alignment. Furthermore, model scale does not predict performance: Mistral-7B (7B parameters) outperforms GPT-OSS-20B (20B parameters) across all metrics. NLI analysis reveals that contradiction rates remain below 7%, suggesting models rarely generate content that directly conflicts with human consensus. These findings challenge the reliability of lexical metrics for evaluating abstractive generation and argue for multi-dimensional evaluation frameworks that capture semantic fidelity beyond surface-level text matching. The RECOM dataset is publicly available at https://anonymous.4open.science/r/recom-D4B0
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
