arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.00873cs.HC
收藏

对齐人类与LLM判断:来自EvalAssist关于任务特定评估和AI辅助评估策略偏好的见解

Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences

Zahra Ashktorab, Michael Desmond, Qian Pan, James M. Johnson, Martin Santillan Cooper, Elizabeth M. Daly, Rahul Nair, Tejaswini Pedapati, Hyo Jin Do, Werner Geyer

AI总结:

本研究通过EvalAssist工具探索LLM评估中直接评估与两两比较策略,发现直接评估更受用户偏好,并提出改进系统交互以支持LLM辅助评估的建议。

摘要:

评估大型语言模型(LLM)的输出需要用户对各种配置下的最佳输出做出关键判断。鉴于数据量庞大,这一过程成本高昂且耗时。LLM越来越多地被用作评估者,以筛选训练数据、评估模型性能或协助人类评估者进行详细评估。为支持这一过程,有效的前端工具对评估至关重要。使用LLM作为评估者的两种常见方法是直接评估和两两比较。在我们与机器学习从业者(n=15)的研究中,每位参与者完成6项任务,共产生131项评估,我们探讨了任务相关因素和评估策略如何影响标准细化及用户感知。研究结果显示,用户通过使标准更具任务特定性、修改判断以及更换评估模型,在直接评估中执行了更多评估。最后,我们提出了系统如何更好地支持LLM辅助评估中交互的建议。

英文摘要:

Evaluation of large language model (LLM) outputs requires users to make critical judgments about the best outputs across various configurations. This process is costly and takes time given the large amounts of data. LLMs are increasingly used as evaluators to filter training data, evaluate model performance or assist human evaluators with detailed assessments. To support this process, effective front-end tools are critical for evaluation. Two common approaches for using LLMs as evaluators are direct assessment and pairwise comparison. In our study with machine learning practitioners (n=15), each completing 6 tasks yielding 131 evaluations, we explore how task-related factors and assessment strategies influence criteria refinement and user perceptions. Findings show that users performed more evaluations with direct assessment by making criteria task-specific, modifying judgments, and changing the evaluator model. We conclude with recommendations for how systems can better support interactions in LLM-assisted evaluations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑