greCAPTCHA：在生成式人工智能下评估理解力作为研究作者身份的证据

中文摘要 AI 辅助

会议、期刊、资助机构、学校及大学正面临大量可能由人工智能生成、却来自表面上为人类作者的投稿，这些作者可能未对其稿件进行充分的人类监督。相应地，评估投稿的机构不再能仅凭投稿作品上的作者姓名可靠地认定其专业能力。为解决这一问题，我们提出了greCAPTCHA，一种有监考的评估方法，通过“验证能力”这一构念来衡量作者对研究手稿的理解程度。我们将“验证能力”定义为批判性评估构成个人对稿件贡献之内容所需的知识与推理能力。greCAPTCHA生成评估多个理解层次的问题，并根据作者的作答提供评估报告。我们使用一个原型实现，对31位研究人员进行了用户研究和半结构化访谈以评估greCAPTCHA。其自动评分预测哪些论文由研究参与者撰写或未由他们撰写，AUC达到0.90。参与者报告了对该系统总体积极的体验，并称赞其对作者理解具有适当的构念效度，同时也提出了在部署前需要做出的重要改进。我们的结果提供了初步证据，表明greCAPTCHA能在有监考条件下评估针对具体手稿的理解力。

英文摘要

Conferences, journals, funders, schools, and universities are struggling with a surge of potentially AI-generated submissions from ostensibly human authors, who may not have exercised sufficient human oversight for their manuscripts. In turn, institutions evaluating submissions can no longer reliably credit expertise based solely on authors' names on submitted work. To address this problem, we propose greCAPTCHA, a proctored assessment approach that measures authors' understanding of research manuscripts via the construct of capacity to verify, which we define as the knowledge and reasoning required to critically assess the contents underlying one's contributions to a manuscript. greCAPTCHA generates questions assessing multiple levels of understanding and provides an evaluative report based on authors' responses. Using a prototype implementation, we conduct a user study and semi-structured interviews with $31$ researchers to evaluate greCAPTCHA. Its automated scores predict which papers were or were not authored by study participants with an AUC of $0.90$. Participants reported positive overall experiences with the system and remarked on the appropriate construct validity for author understanding, while also suggesting important changes to be made before deployment. Our results provide initial evidence that greCAPTCHA can assess manuscript-specific understanding under proctored conditions.

发表机构

• Carnegie Mellon University（卡内基梅隆大学）

机构由 AI 辅助整理，请以论文原文为准。