大语言模型是否回答了问题？一种用于越狱评估的语义重建游戏
Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation
AI总结：
本研究针对LLM越狱评估中现有评估器的误差与ASR变异问题，提出语义重建游戏，该方法在三个现成评估器中对人工标签平均误差最低，可降低ASR变异度且优先避免误报。
摘要：
针对大语言模型（LLM）的越狱攻击采用攻击成功率（ASR）评估，该指标依赖于正确识别满足禁止请求的响应。现有评估器可能将拒绝或主题相关响应误判为成功攻击。我们表明，仅更改评估LLM就可使报告的ASR变化高达82个百分点。我们提出语义重建游戏，一种无需训练的评估器，它将合规性判断分解为受约束的语义比较：仅给定响应，LLM重建其实质性内容所回答的请求，排除仅重复或拒绝的信息；该游戏将此重建与类似诱饵请求比较，判断是否匹配原始请求，当两个候选均不匹配时允许弃权（不执行）。在三个现成LLM评估器上，该游戏在所有评估方法中对人工标签的平均误差最低，且降低了报告ASR的变异度；它优先避免误报，即便以错过部分成功越狱为代价。
英文摘要：
Jailbreak attacks against large language models (LLMs) are evaluated using attack success rate (ASR), which depends on correctly identifying responses that fulfill prohibited requests. Existing evaluators can mistake refusals or topically related responses for successful attacks. We show that changing only the judge LLM can alter reported ASR by up to $82$ percentage points. We introduce the semantic reconstruction game, a training-free evaluator that decomposes compliance judgments into constrained semantic comparisons. Given only a response, an LLM reconstructs the request answered by its substantive content, excluding information merely repeated or refused. The game compares this reconstruction against similar decoy requests to determine whether it matches the original request, allowing abstention when neither candidate matches. Across three off-the-shelf LLM judges, the game achieves the lowest mean error against human labels among evaluated methods and reduces variation in reported ASR. It prioritizes avoiding false positives, even at the cost of missing some successful jailbreaks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
