Cross-lingual versatility has become a defining capability of Large Language Models (LLMs). However, fine-tuning these models often induces negative interference, where gains in one language come at another's expense. To address this, we cast multilingual fine-tuning as a multi-objective optimization (MOO) problem. Specifically, we introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally, within the parameter buckets of distributed training. This enables conflict-aware updates without ever materializing a full gradient vector for any objective, and improves multilingual performance over standard fine-tuning by up to $4.3$ points. Theoretically, we define bucketwise Pareto stationarity, show it is necessary for Pareto optimality under any fixed partition, and prove descent and convergence for representative Bucket-Level variants. Our work brings conventional MOO within reach for multilingual LLMs at a far lower cost. More broadly, Bucket-Level MOO is agnostic to both the objectives and the update rule, readily extending to other multi-task learning at scale.
Cross-linguistic disagreement as a conflict of semantic alignment norms in multilingual AI~Linguistic Diversity as a Problem for Philosophy, Cognitive Science, and AI~
Japan Advanced Institute of Science and Technology(日本先进科学技术学院);Victoria University of Wellington(惠灵顿维多利亚大学);Macquarie University(麦考瑞大学);Nanzan University(南山大学);Kyoto University(京都大学)
Surgical Knowledge Rewrite in Compact LLMs: An 'Unlearn-then-Learn' Strategy with ($IA^3$) for Localized Factual Modulation and Catastrophic Forgetting Mitigation
Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values
School of Information Science and Technology, Beijing Foreign Studies University(北京外国语大学信息科学与技术学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beihang University(北京航空航天大学)
From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University(上海智能信息处理关键实验室,复旦大学计算机学院);Shanghai Collaborative Innovation Center on Intelligent Visual Computing(上海智能视觉计算协同创新中心);Singapore Management University(新加坡管理大学)
Where Reasoning Diverges: Localized Multi-Agent Debate for Multi-Hop Question Answering
The Chinese University of Hong Kong(香港中文大学);Nagoya University(名古屋大学);Institute of Science Tokyo(东京科学大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Amplify Initiative: Building A Localized Data Platform for Globalized AI
Google Research USA(谷歌研究美国分公司);Makerere University(Makerere大学);Google Research India(谷歌研究印度);Jommo Kenyatta University Agriculture & Technology(乔姆摩肯雅塔大学农业与技术);Hutzpa Innovations Nigeria(Hutzpa创新尼日利亚);Malawi University of Science & Technology(马拉维科学技术大学)