arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.16740cs.HC
收藏

在实验室中评估生成式AI:方法学挑战与指南

Evaluating Generative AI in the Lab: Methodological Challenges and Guidelines

BMW Group(宝马集团);University of Stuttgart(斯图加特大学);LMU Munich(慕尼黑大学);TU Dortmund University(多特蒙德工业大学)

Hyerim Park, Khanh Huynh, Malin Eiband, Jeremy Dillmann, Sven Mayer, Michael Sedlmair

AI总结:

本文探讨了在实验室环境中评估生成式AI的方法学挑战,提出了五个指南和十八项建议,以帮助研究人员设计更透明和稳健的评估研究。

摘要:

生成式AI(GenAI)系统本质上是非确定性的,即使对于相同的输入也会产生多样的输出。尽管这种变异性是其吸引力的核心,但它挑战了传统人机交互(HCI)评估实践,这些实践通常假设系统行为一致且可预测。因此,在这种条件下设计受控的实验室研究仍然是一个关键的方法学挑战。我们对四个基于实验室的用户研究进行了反思性的多案例分析,这些研究涉及整合了生成式AI原型的系统,涵盖车载对话助手系统和用于设计工作流程的图像生成工具。通过跨案例反思和所有研究阶段的主题分析,我们识别出五个方法学挑战,并提出十八项实践导向的建议,分为五个指南。这些挑战代表了方法学构念,它们要么被GenAI的随机性放大、重新定义或引入:(C1)依赖于熟悉的交互模式,(C2)保真度与控制权之间的权衡,(C3)反馈与信任,(C4)可用性评估的空白,以及(C5)界面和系统问题之间的解释模糊性。我们的指南通过策略来解决这些挑战,例如将上手流程重新定义为帮助参与者管理不可预测性,扩展评估以包含信任和意图对齐等构念,并记录系统事件,包括幻觉和延迟,以支持透明的分析。这项工作贡献了(1)对GenAI的随机性如何扰乱基于实验室的HCI评估的方法学反思,以及(2)十八项建议,帮助研究人员在受控环境中设计更加透明、稳健和可比的研究。

英文摘要:

Generative AI (GenAI) systems are inherently non-deterministic, producing varied outputs even for identical inputs. While this variability is central to their appeal, it challenges established HCI evaluation practices that typically assume consistent and predictable system behavior. Designing controlled lab studies under such conditions therefore remains a key methodological challenge. We present a reflective multi-case analysis of four lab-based user studies with GenAI-integrated prototypes, spanning conversational in-car assistant systems and image generation tools for design workflows. Through cross-case reflection and thematic analysis across all study phases, we identify five methodological challenges and propose eighteen practice-oriented recommendations, organized into five guidelines. These challenges represent methodological constructs that are either amplified, redefined, or newly introduced by GenAI's stochastic nature: (C1) reliance on familiar interaction patterns, (C2) fidelity-control trade-offs, (C3) feedback and trust, (C4) gaps in usability evaluation, and (C5) interpretive ambiguity between interface and system issues. Our guidelines address these challenges through strategies such as reframing onboarding to help participants manage unpredictability, extending evaluation with constructs such as trust and intent alignment, and logging system events, including hallucinations and latency, to support transparent analysis. This work contributes (1) a methodological reflection on how GenAI's stochastic nature unsettles lab-based HCI evaluation and (2) eighteen recommendations that help researchers design more transparent, robust, and comparable studies of GenAI systems in controlled settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑