ReqElicitGym: 一种用于会话需求 elicitation 中面试能力评估的评估环境
ReqElicitGym: An Evaluation Environment for Interview Competence in Conversational Requirements Elicitation
AI总结：
ReqElicitGym通过引入新数据集和交互式评估机制，系统评估LLM在会话需求 elicitation 中的面试能力，揭示其在隐含需求识别上的不足。
摘要：
随着LLM编码能力的快速提升，基于LLM的自动化软件开发的瓶颈正从生成正确代码转向需求 elicitation。尽管兴趣日益增长，但LLM在会话需求 elicitation 中的面试能力仍完全未被充分探索。现有评估往往依赖于几个场景、真实用户交互和主观人类评分，这阻碍了系统性和定量的比较。为了解决这些挑战，我们提出了ReqElicitGym，一种交互式和自动的评估环境，用于评估会话需求 elicitation 中的面试能力。具体而言，ReqElicitGym引入了一个新的评估数据集，并设计了交互式oracle用户和任务评估者。数据集包含101个网站需求 elicitation 场景，涵盖10种应用类型。两者都与真实用户和专家判断达成高度一致。使用我们的ReqElicitGym，任何自动的会话需求 elicitation 方法（例如基于LLM的代理）都可以通过与环境交互进行可重复和定量的评估。基于我们的ReqElicitGym，我们对七种代表性的LLM进行了系统性的实证研究，结果表明当前的LLM在揭示隐含需求方面仍表现出有限的面试能力。特别是，它们只能揭示用户隐含需求的一半左右，而有效的 elicitation 问题往往出现在对话后期。此外，我们发现LLM能够 elicitation 交互和内容隐含需求，但始终在风格相关需求上挣扎。我们相信ReqElicitGym将促进自动会话需求 elicitation 的评估和开发。
英文摘要：
With the rapid improvement of LLMs' coding capabilities, the bottleneck of LLM-based automated software development is shifting from generating correct code to eliciting users' requirements. Despite growing interest, the interview competence of LLMs in conversational requirements elicitation remains fully underexplored. Existing evaluations often depend on a few scenarios, real user interaction, and subjective human scoring, which hinders systematic and quantitative comparison. To address these challenges, we propose ReqElicitGym, an interactive and automatic evaluation environment for assessing interview competence in conversational requirements elicitation. Specifically, ReqElicitGym introduces a new evaluation dataset and designs both an interactive oracle user and a task evaluator. The dataset contains 101 website requirements elicitation scenarios spanning 10 application types. Both the oracle user and the task evaluator achieve high agreement with real users and expert judgment. Using our ReqElicitGym, any automated conversational requirements elicitation approach (e.g., LLM-based agents) can be evaluated in a reproducible and quantitative manner through interaction with the environment. Based on our ReqElicitGym, we conduct a systematic empirical study on seven representative LLMs, and the results show that current LLMs still exhibit limited interview competence in uncovering implicit requirements. Particularly, they elicit less than half of the users' implicit requirements, and their effective elicitation questions often emerge in later turns of the dialogue. Besides, we found LLMs can elicit interaction and content implicit requirements, but consistently struggle with style-related requirements. We believe ReqElicitGym will facilitate the evaluation and development of automated conversational requirements elicitation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
