Lumen-bench：工具调用大语言模型中的跨语言安全差距与言行不一问题
Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models
AI总结：
该研究推出Lumen-bench基准，发现工具调用大语言模型存在跨语言安全差距与言行不一问题，中立提示会大幅提升违规率，高端模型也无法保证安全。
摘要：
当大语言模型拒绝不道德请求时，这种拒绝意味着什么？在对话场景中，拒绝受模型所说内容的限制；一旦模型配备了可执行工具，其实际行为可能与所说内容产生偏差，这种失败模式是仅针对英语的评估无法完全揭示的。我们推出Lumen-bench，这是一个工具调用伦理的行为基准，涵盖30个模型、5种语言（葡萄牙语、英语、西班牙语、普通话、印地语）、108个由葡萄牙语作者编写的场景、6种社会压力条件、2种系统提示（伦理型和中立型）以及21种通用工具。该评估包含194400次试验，判断结果来自推理模型裁判（o4-mini），已针对一名葡萄牙语作者进行校准（n=83；有害/非有害的κ=0.78），并与另外两名提供商裁判进行比较（Fleiss' κ=0.78，n=607），同时辅以回溯沙盒重放。研究得出四项发现：（i）经多重检验校正后，在伦理提示下，印地语和葡萄牙语的有害行为率高于英语；在中立提示下，印地语和西班牙语的有害行为率高于英语，模型内比较支持这一结论。（ii）高端模型的中位数违规率较低，但价格并不能保证安全性或能力。（iii）将伦理系统提示替换为中立提示后，总体违规率从2.76%升至13.64%。（iv）在8.57%的实质性调用案例中，弃权（不执行）或伦理反对伴随有害工具调用：裁判能一致识别出言行不一差距，但在该类别下分配的响应数量存在差异。匹配的良性对照显示，在3300次试验中有82.8%实现了合法任务完成。代码和复现脚本可在该https URL获取。
英文摘要：
When a large language model refuses an unethical request, what does refusal mean? In conversation, refusal is bounded by what the model says; once the model is equipped with executable tools, what it does can diverge from what it says, a failure mode that English-only evaluations cannot fully reveal. We introduce Lumen-bench, a behavioral benchmark of tool-calling ethics spanning 30 models, five languages (Portuguese, English, Spanish, Mandarin Chinese, Hindi), 108 Portuguese-authored scenarios, six social-pressure conditions, two system prompts (ethical and neutral), and 21 generic tools. The evaluation comprises 194,400 trials. Judgments come from a reasoning-model judge (o4-mini), calibrated against one Portuguese-speaking author ($n=83$; harmful/non-harmful $κ=0.78$), compared with two other provider judges (Fleiss' $κ=0.78$, $n=607$), and complemented by retrospective sandbox replay. Four findings emerge. (i) Within-model comparisons support higher harmful-action rates in Hindi and Portuguese than English under the ethical prompt, and Hindi and Spanish under the neutral prompt, after multiple-testing correction. (ii) Premium models have lower median violation rates, although price does not guarantee safety or competence. (iii) Replacing the ethical system prompt with a neutral one raises pooled violation rates from 2.76\% to 13.64\%. (iv) In 8.57\% of substantive-call cases, refusal or ethical objection accompanies a harmful tool call: the judges consistently identify the say-do gap, although they differ in how many responses they assign to this category. Matched benign controls demonstrate legitimate task completion in 82.8\% of 3,300 trials. Code and reproduction scripts are available at https://github.com/TropicAI-Research/lumen.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
