面向人道主义NLP的指令微调语言模型的跨语言稳定性与偏差
Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP
AI总结：
本研究首次系统比较商业与开源权重LLM在七种语言的人权侵犯检测中的表现，提出四种跨语言可靠性指标，发现指令对齐而非模型规模决定跨语言稳定性，为资源受限组织提供部署指导。
摘要：
人道主义组织面临一个关键抉择：投资于昂贵的商业API，还是依赖免费的开源权重模型进行多语种人权监测。虽然商业系统提供可靠性，但开源权重替代方案缺乏经验验证——尤其是在冲突地区常见的低资源语言方面。本文首次对商业和开源权重大语言模型（LLMs）在七种语言的人权侵犯检测任务上进行了系统性比较，量化了资源受限组织面临的成本-可靠性权衡。在78,000次多语种推理中，我们评估了六个模型——四个指令对齐模型（Claude-Sonnet-4、DeepSeek-V3、Gemini-Flash-2.0、GPT-4.1-mini）和两个开源权重模型（LLaMA-3-8B、Mistral-7B）——使用标准分类指标和新的跨语言可靠性度量：校准偏差（CD）、决策偏差（B）、语言鲁棒性评分（LRS）和语言稳定性评分（LSS）。结果表明，对齐而非规模决定稳定性：对齐模型在类型学上遥远和低资源语言（如林加拉语、缅甸语）中保持近乎不变的准确性和均衡校准，而开源权重模型表现出显著的提示语言敏感性和校准漂移。这些发现证明多语种对齐能够实现语言无关的推理，并为在预算限制与多语种部署可靠性之间寻求平衡的人道主义组织提供实践指导。
英文摘要：
Humanitarian organizations face a critical choice: invest in costly commercial APIs or rely on free open-weight models for multilingual human rights monitoring. While commercial systems offer reliability, open-weight alternatives lack empirical validation -- especially for low-resource languages common in conflict zones. This paper presents the first systematic comparison of commercial and open-weight large language models (LLMs) for human-rights-violation detection across seven languages, quantifying the cost-reliability trade-off facing resource-constrained organizations. Across 78,000 multilingual inferences, we evaluate six models -- four instruction-aligned (Claude-Sonnet-4, DeepSeek-V3, Gemini-Flash-2.0, GPT-4.1-mini) and two open-weight (LLaMA-3-8B, Mistral-7B) -- using both standard classification metrics and new measures of cross-lingual reliability: Calibration Deviation (CD), Decision Bias (B), Language Robustness Score (LRS), and Language Stability Score (LSS). Results show that alignment, not scale, determines stability: aligned models maintain near-invariant accuracy and balanced calibration across typologically distant and low-resource languages (e.g., Lingala, Burmese), while open-weight models exhibit significant prompt-language sensitivity and calibration drift. These findings demonstrate that multilingual alignment enables language-agnostic reasoning and provide practical guidance for humanitarian organizations balancing budget constraints with reliability in multilingual deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
