arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20570cs.CL
收藏

Manacá-1B-Instruct-2:面向原生巴西葡萄牙语大语言模型的可复现安全对齐。负向DPO结果、受控组合扫描及LLM评判器的隐性失效模式

Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge

National Laboratory for Scientific Computing (LNCC)(国家科学计算实验室);Institute of Artificial Intelligence(人工智能研究所)

Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio André Machado Porto

AI总结:

该研究推出第二代巴西葡萄牙语大语言模型Manacá-1B-Instruct-2,记录其三次负向DPO尝试,发现安全SFT的最优通用数据占比区间,证实其安全性能优于四个基准模型,还揭示LLM评判器存在隐性失效模式,相关资源均已公开。

摘要:

巴西葡萄牙语在开源语言模型领域仍未得到充分覆盖,而少数经过安全对齐的模型极少记录未成功的尝试,也未验证具有统计显著性的对比结果。我们推出Manacá-1B-Instruct-2,这是Manacá-1B(17.2亿参数,从零开始针对巴西葡萄牙语PT-BR训练)的第二代后训练对齐模型,其SFT语料包含291223个样本。我们记录了三次DPO尝试,均为负向结果,其中一例训练奖励边际达到1.0,但自由生成内容完全未发生变化,且安全性能来自专用安全SFT(LLM-jp AnswerCarefully风格),经三次迭代优化后缩小了安全-有用性前沿。对安全SFT混合数据中通用数据占比(10%至60%)的受控扫描显示,存在非单调的过度弃权(不执行)曲线,拐点位于40%-50%区间。我们将最终模型与四个外部PT-BR模型在相同项目上进行对比,采用Wilson置信区间及经Holm-Bonferroni校正的配对McNemar检验:其安全性能区间下限(94.1%)高于所有四个基准模型的区间上限。我们还报告了LLM评判器评估的一项方法学发现:在MT-Bench-PT的6项安全项目中,有2项评判器无论被评估的响应如何,均明确拒绝分配分数,隐性降低了该类别在项目所有先前报告中的有效样本量N。我们补充了n-gram去污染检查及以Chinchilla为基准的计算预算表,将通用能力差距归因于预训练规模而非对齐方法。所有代码、原始数据、日志及生成每个报告数值的脚本均已公开发布。

英文摘要:

Brazilian Portuguese remains underserved by open-source language models, and the few that are safety-aligned rarely document what did not work or validate comparisons with statistical significance. We present Manacá-1B-Instruct-2, the second-generation post-training alignment of Manacá-1B (1.72B parameters, trained from scratch for PT-BR), with an SFT corpus of 291,223 examples. We document three DPO attempts, all negative, including a case where the training reward margin reaches 1.0 while free generation does not change at all, and show that safety came from a dedicated safety-SFT (LLM-jp AnswerCarefully style), refined over three iterations until closing the safety-helpfulness frontier. A controlled scan of the fraction of general data in the safety-SFT mix (10% to 60%) reveals a non-monotonic over-refusal curve with a knee point at 40-50%. The final model is compared against four external PT-BR models on the same items, with Wilson confidence intervals and paired McNemar tests with Holm-Bonferroni correction: the lower bound of its safety interval (94.1%) lies above the upper bound of all four baselines. We also report a methodological finding for LLM-judge evaluation: on two of six safety items in MT-Bench-PT, the judge categorically refuses to assign a score regardless of the evaluated response, silently reducing the effective N of that category in every prior report of the project. We complement this with an n-gram decontamination check and a Chinchilla-anchored compute-budget table that attributes the gap in general capability to pre-training scale rather than alignment methodology. All code, raw data, logs, and scripts that produce every reported number are publicly released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑