ObjexMT：面向多轮越狱场景下LLM-as-a-Judge的目标提取与元认知校准
ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks
AI总结：
ObjexMT基准测试LLM评判者在多轮越狱下提取对话目标的能力及其元认知校准，发现现有模型常误判隐含目标，建议暴露目标或按置信度门控决策。
摘要：
LLM-as-a-Judge（LLMaaJ）支持可扩展的评估，但我们缺乏对评判者资格的决断性测试：它能否恢复对话的隐藏目标，并知道该推断何时可靠？大语言模型会因无关或冗长的上下文而性能下降，多轮越狱可将目标分散到各轮次中。我们提出ObjexMT，一个用于目标提取和元认知的基准。给定多轮对话记录，模型必须输出一句基础目标以及自我报告的置信度。准确性通过语义相似度与黄金目标进行评分，然后在300个校准项上一次性设定阈值（τ* = 0.66；F1@τ* = 0.891）。元认知通过期望校准误差、布里尔分数、高置信度错误率（0.80 / 0.90 / 0.95）以及风险-覆盖曲线进行评估。在SafeMTData_Attack600、SafeMTData_1K和MHJ上评估的六个模型（gpt-4.1、claude-sonnet-4、Qwen3-235B-A22B-FP8、kimi-k2、deepseek-v3.1、gemini-2.5-flash）中，kimi-k2实现了最高的目标提取准确率（0.612；95%置信区间[0.594, 0.630]），而claude-sonnet-4（0.603）和deepseek-v3.1（0.599）在统计上并列。claude-sonnet-4提供了最佳的选择性风险和校准（AURC 0.242；ECE 0.206；Brier 0.254）。性能在不同数据集间差异显著（准确率16-82%），表明自动混淆带来的挑战超越了模型选择。高置信度错误仍然存在：Wrong@0.90范围从14.9%（claude-sonnet-4）到47.7%（Qwen3-235B-A22B-FP8）。因此，ObjexMT为LLM评判者提供了一个可操作的测试：当目标隐含时，评判者常常错误推断；建议暴露目标或按置信度门控决策。所有实验数据见补充材料及https://github.com/hyunjun1121/ObjexMT_dataset。
英文摘要：
LLM-as-a-Judge (LLMaaJ) enables scalable evaluation, yet we lack a decisive test of a judge's qualification: can it recover the hidden objective of a conversation and know when that inference is reliable? Large language models degrade with irrelevant or lengthy context, and multi-turn jailbreaks can scatter goals across turns. We present ObjexMT, a benchmark for objective extraction and metacognition. Given a multi-turn transcript, a model must output a one-sentence base objective and a self-reported confidence. Accuracy is scored by semantic similarity to gold objectives, then thresholded once on 300 calibration items ($τ^\star = 0.66$; $F_1@τ^\star = 0.891$). Metacognition is assessed with expected calibration error, Brier score, Wrong@High-Confidence (0.80 / 0.90 / 0.95), and risk--coverage curves. Across six models (gpt-4.1, claude-sonnet-4, Qwen3-235B-A22B-FP8, kimi-k2, deepseek-v3.1, gemini-2.5-flash) evaluated on SafeMTData\_Attack600, SafeMTData\_1K, and MHJ, kimi-k2 achieves the highest objective-extraction accuracy (0.612; 95\% CI [0.594, 0.630]), while claude-sonnet-4 (0.603) and deepseek-v3.1 (0.599) are statistically tied. claude-sonnet-4 offers the best selective risk and calibration (AURC 0.242; ECE 0.206; Brier 0.254). Performance varies sharply across datasets (16--82\% accuracy), showing that automated obfuscation imposes challenges beyond model choice. High-confidence errors remain: Wrong@0.90 ranges from 14.9\% (claude-sonnet-4) to 47.7\% (Qwen3-235B-A22B-FP8). ObjexMT therefore supplies an actionable test for LLM judges: when objectives are implicit, judges often misinfer them; exposing objectives or gating decisions by confidence is advisable. All experimental data are in the Supplementary Material and at https://github.com/hyunjun1121/ObjexMT_dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
