多语言大模型任务执行中的语言角色解耦
Disentangling Language Roles in Multilingual LLM Task Execution
AI总结：
提出MTM-Bench基准，通过完全交叉设计解耦指令、内容和响应三种语言角色，评估多语言LLM的任务执行能力，发现响应语言角色是性能下降的主要因素。
摘要：
多语言大模型在指令、源内容和所需响应语言不一致时被越来越多地使用。现有基准扩展了多语言指令跟随评估，但很少在完全交叉设计中隔离这三种角色。我们引入了MTM-Bench，一个用于语言条件任务执行的控制基准，其中每个实例由三元组 \((L_{\text{instr}}, L_{\text{content}}, L_{\text{resp}})\) 定义。在英语、西班牙语和中文中，MTM-Bench枚举了所有27个三元组，每个模型包含2,430个实例，涵盖语义反转、最终状态提取和带更新实现的语言纯度。我们使用分解指标评估了20个前沿和开源权重LLM，包括语义正确性、目标语言遵循度、约束满足度、污染比率和联合成功率，并通过针对性的人工审计验证评分。完全交叉设计揭示了性能下降是由语言在任务结构中扮演的角色组织的，而不仅仅是语言不匹配的数量。响应语言角色是变化的主要轴，单个响应槽不匹配导致了大部分性能下降。仅响应不匹配与完全不匹配的比较表明，不匹配数量不是困难的单调预测因子，模型级别的排序在不同系统间变化。任务族通过不同的通道失败，表明语义正确性本身并不能捕捉可靠的多语言任务执行。
英文摘要：
Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet \((L_{\text{instr}}, L_{\text{content}}, L_{\text{resp}})\). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2{,}430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
