arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.06997cs.CYcs.AI
收藏

目标设定理论在LLM驱动评估中的局限性

The Limits of Goal-Setting Theory in LLM-Driven Assessment

Birla Institute of Technology(巴尔塔理工学院)

Mrityunjay Kumar

AI总结:

本研究通过受控实验发现,增加提示特异性并未提升LLM评估的一致性,挑战了目标设定理论在LLM中的应用,并指出需增强模型鲁棒性与输入整合。

摘要:

许多用户使用一种将系统视为类人的心智模型与ChatGPT等AI工具交互,我们称之为模型H。根据目标设定理论,目标的特异性增加应减少绩效方差。如果模型H成立,那么用更详细的指令提示聊天机器人应导致更一致的评估行为。本文通过一项受控实验检验了这一假设,实验中ChatGPT使用四个特异性递增的提示评估了29份学生提交物。我们使用重测信度(Cohen's Kappa)在重复运行中测量一致性。与预期相反,绩效并未随提示特异性增加而一致改善,绩效方差基本保持不变。这些发现挑战了LLM像人类评估者一样行为的假设,并强调了在未来的模型开发中需要更强的鲁棒性和改进的输入整合。

英文摘要:

Many users interact with AI tools like ChatGPT using a mental model that treats the system as human-like, which we call Model H. According to goal-setting theory, increased specificity in goals should reduce performance variance. If Model H holds, then prompting a chatbot with more detailed instructions should lead to more consistent evaluation behavior. This paper tests that assumption through a controlled experiment in which ChatGPT evaluated 29 student submissions using four prompts with increasing specificity. We measured consistency using intra-rater reliability (Cohen's Kappa) across repeated runs. Contrary to expectations, performance did not improve consistently with increased prompt specificity, and performance variance remained largely unchanged. These findings challenge the assumption that LLMs behave like human evaluators and highlight the need for greater robustness and improved input integration in future model development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑