AA-Omniscience：评估大型语言模型中的跨领域知识可靠性
AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models
AI总结：
该研究针对大型语言模型跨领域知识可靠性问题，提出AA-Omniscience基准，通过6000个问题评估事实回忆与知识校准能力，揭示模型弱点并发现领域性能差异，为模型选择提供依据。
摘要：
现有语言模型评估主要衡量通用能力，但跨领域可靠使用这些模型需要事实准确性和对知识缺口的识别。我们引入AA-Omniscience基准，旨在通过6000个问题衡量事实回忆与知识校准能力。问题源自权威学术和行业来源，覆盖6个领域内42个经济相关主题。评估测量模型的全知指数（Omniscience Index），这是一个范围为-100至100的有界指标，衡量事实回忆能力，会联合惩罚幻觉并奖励不确定时的弃权（不执行），0分相当于正确与错误回答数量相等的模型。在评估的模型中，Claude 4.1 Opus获得最高分（4.8），是仅有的三个得分高于0的模型之一。这些结果揭示前沿模型在事实性和校准方面持续存在弱点。不同领域的性能存在差异，来自三个不同研究实验室的模型在六个领域中分别领先。这种性能差异表明，在知识重要的任务中，应根据用例需求选择模型，而非仅依据通用性能。
英文摘要：
Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
