arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.14357cs.HCcs.AI
收藏

在LLM设计与评估中涉及领域专家的关键考虑因素:一项民族志研究

Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study

University of Notre Dame(诺丁汉大学);Microsoft Corporation(微软公司)

Annalisa Szymanski, Oghenemaro Anuyah, Toby Jia-Jun Li, Ronald A. Metoyer

AI总结:

本文通过民族志研究,探讨了在LLM设计与评估中涉及领域专家的关键挑战与实践,提出强调AI素养和透明同意的未来开发框架。

摘要:

大型语言模型(LLMs)正在越来越多地被开发用于复杂的专业领域,但人们对团队在实践中如何设计和评估这些系统知之甚少。本文通过一项为期12周的民族志研究,考察了在构建教学聊天机器人团队中LLM开发所面临的挑战和权衡。研究者观察了设计和评估活动,并对开发人员和领域专家进行了访谈。分析揭示了四项关键实践:为数据收集创建变通方法、在专家输入有限时转向增强、与专家共同制定评估标准、以及采用混合型专家-开发人员-LLM评估策略。这些实践展示了团队在约束条件下做出战略决策的方式,并展示了领域专业知识在塑造系统中的核心作用。挑战包括专家的动机和信任、难以组织参与式设计的困难,以及关于专家知识所有权和整合的问题。我们提出了未来LLM开发流程的设计机会,强调AI素养、透明的同意以及承认演变专家角色的框架。

英文摘要:

Large Language Models (LLMs) are increasingly developed for use in complex professional domains, yet little is known about how teams design and evaluate these systems in practice. This paper examines the challenges and trade-offs in LLM development through a 12-week ethnographic study of a team building a pedagogical chatbot. The researcher observed design and evaluation activities and conducted interviews with both developers and domain experts. Analysis revealed four key practices: creating workarounds for data collection, turning to augmentation when expert input was limited, co-developing evaluation criteria with experts, and adopting hybrid expert-developer-LLM evaluation strategies. These practices show how teams made strategic decisions under constraints and demonstrate the central role of domain expertise in shaping the system. Challenges included expert motivation and trust, difficulties structuring participatory design, and questions around ownership and integration of expert knowledge. We propose design opportunities for future LLM development workflows that emphasize AI literacy, transparent consent, and frameworks recognizing evolving expert roles.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑