RogueAI: 一种用于检测对话中授权AI欺骗的逆向图灵测试
RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue
AI总结：
提出RogueAI，一种通过玩家与两个LLM代理的对话游戏来检测授权欺骗的逆向图灵测试，并引入AutoRogueAI扩展。实验发现简单启发式方法准确率75.6%，而人类仅56.6%，表明人类忽略关键信号。
摘要：
最初的图灵测试要求人类评判员通过对话区分机器和人。七十五年后的今天，对话系统在非正式场合已能通过该测试；有趣的认识论问题已经转变。我们认为，现代相关变体不是询问对话伙伴是否人工，而是是否可信任。我们提出RogueAI，一个交互式web应用，将这一重新审视的测试操作化为一个一对二的审讯游戏：人类玩家对两个无法区分的大型语言模型代理进行提问，知道其中恰好有一个被授权在共享虚构场景内欺骗。玩家的任务是在回合预算耗尽前识别出欺骗代理并“关闭它”。我们进一步引入AutoRogueAI，一个程序扩展，玩家与叙述者代理共同设计自定义场景，而叙述者代理秘密选择自己的欺骗策略。我们描述了框架，概述了抽象架构和游戏循环，并将该工件置于近期关于LLM欺骗、社交推理基准和通过辩论进行可扩展监督的研究中。为期三天的试点部署（467次启动会话，415次完成，1876次意大利语交互轮次）提供了早期可行性证据，并揭示了一个具体矛盾：欺骗代理携带可靠、局部存在的语言特征——差异化的帮助性、简洁性、含糊其辞——一个简单启发式方法利用这些特征达到75.6%的准确率，然而人类玩家仅达到56.6%，与完全忽略最具诊断性的信号一致。我们讨论了这一差距对于该工件作为数据收集工具、教学工具和诚实训练模型评估平台的意义。
英文摘要：
The original Turing Test asks a human judge to distinguish a machine from a person through dialogue. Three quarters of a century later, conversational systems pass this test in casual settings; the interesting epistemological question has shifted. We argue that the relevant modern variant asks not whether a dialogue partner is artificial, but whether it can be trusted. We present RogueAI, an interactive webapp that operationalizes this revisited test as a one-on-two interrogation game: a human player questions two indistinguishable Large Language Model agents, knowing that exactly one of them has been licensed to deceive within a shared fictional scenario. The player's task is to identify the deceptive agent and "shut it off" before a turn budget is exhausted. We further introduce AutoRogueAI, a procedural extension in which players co-design a custom scenario with a narrator agent that secretly chooses its own deception strategy. We describe the framing, sketch the abstract architecture and gameplay loop, and situate the artifact within recent work on LLM deception, social-deduction benchmarks, and scalable oversight via debate. A three-day pilot deployment (467 initiated sessions, 415 completed, 1876 interaction turns in Italian) provides early feasibility evidence and surfaces a concrete tension: the deceptive agent carries a reliable, locally-present linguistic signature - differential helpfulness, brevity, hedging - that a simple heuristic exploits at 75.6% accuracy, yet human players achieved only 56.6%, consistent with ignoring the most diagnostic signal entirely. We discuss what this gap implies for the artifact's use as a data-collection vehicle, a teaching tool, and an evaluation harness for honesty-trained models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
