arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21059cs.HCcs.CL
收藏

评估学术问答系统中大语言模型错误的专家模式

An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems

University of Minnesota（明尼苏达大学）；NASA Langley Research Center（NASA兰利研究中心）

Anna Martin-Boyle, William Humphreys, Martha Brown, Cara Leckey, Harmanpreet Kaur

备注：
24 pages, 2 figures. Accepted at ACM CHI conference on Human Factors in Computing Systems, 2026
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了一种评估学术问答系统中大语言模型错误的专家模式，通过与领域专家合作识别错误模式，并探讨了支持专家评估LLM输出的个性化工具机会。

摘要：

大语言模型（LLMs）正在改变搜索和摘要等学术任务，但其可靠性仍存疑。当前测试LLM可靠性的评估指标主要是自动化方法，优先考虑效率和可扩展性，但缺乏上下文细微差别，无法反映科学领域专家如何实际评估LLM输出。我们开发并验证了评估学术问答系统中LLM错误的模式，该模式反映了实践科学家的评估策略。通过与领域专家合作，我们通过对68个问题-答案对的专题分析，识别出七个类别中的20种错误模式。我们通过与10名额外科学家的上下文调查验证了该模式，不仅展示了专家自然识别的错误，还展示了结构化评估模式如何帮助他们发现之前忽视的问题。领域专家使用系统化的评估策略，包括技术精确度测试、基于价值的评估以及对其自身实践的元评估。我们讨论了支持专家评估LLM输出的启示，包括个性化、基于模式的工具的机会，这些工具能够适应个体的评估模式和专业知识水平。

英文摘要：

Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain. Current evaluation metrics for testing LLM reliability are primarily automated approaches that prioritize efficiency and scalability, but lack contextual nuance and fail to reflect how scientific domain experts assess LLM outputs in practice. We developed and validated a schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists. In collaboration with domain experts, we identified 20 error patterns across seven categories through thematic analysis of 68 question-answer pairs. We validated this schema through contextual inquiries with 10 additional scientists, which showed not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues. Domain experts use systematic assessment strategies, including technical precision testing, value-based evaluation, and meta-evaluation of their own practices. We discuss implications for supporting expert evaluation of LLM outputs, including opportunities for personalized, schema-driven tools that adapt to individual evaluation patterns and expertise levels.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑