arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.25342cs.CL
收藏

MATO: 面向大语言模型的多目标个性化对齐与测试时优化

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Monash University(墨尔本大学);Defence Science and Technology Group, Australia(澳大利亚国防科学与技术集团)

Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen, Junae Kim, Gholamreza Haffari, Dinh Phung

AI总结:

提出MATO框架,通过测试时优化在解码过程中动态调整多目标权重,无需训练或外部奖励模型,实现大语言模型与用户多样化偏好的对齐。

摘要:

将大语言模型与多样且多方面的用户偏好对齐是个性化AI系统的基本挑战。现有的多目标对齐方法要么依赖昂贵的训练,要么需要为每个偏好预训练奖励模型,这使得它们难以适应不断变化的偏好。基于提示的个性化提供了一种无需训练的替代方案,但仅靠提示通常提供有限的可操控性,因为大语言模型可能过度强调或忽略某些偏好,并且在冲突出现时无法让用户可靠地控制不同目标的相对重要性,导致对齐效果欠佳。在本文中,我们介绍了MATO,一种无需训练的多目标个性化对齐与测试时优化框架。MATO将个性化表述为一个测试时优化问题,在解码过程中通过可控权重引导多个目标的相对重要性,无需修改模型参数或需要外部奖励模型。具体来说,奖励发现模块直接从骨干大语言模型中恢复针对自然语言指定的多种目标的偏好奖励,而权重优化模块根据用户的初始偏好和部分生成的响应动态调整目标权重,以在生成过程中平衡相互竞争的目标。得到的奖励和权重共同指导对令牌分布的在线优化过程,从而更好地与目标对齐。在多个数据集和骨干大语言模型上的大量实验表明,MATO始终优于强基线,实现了帕累托改进的多目标对齐和更强的可操控性。这些结果凸显了测试时优化作为可扩展、可控且模型无关的个性化对齐的一个有前景的方向。

英文摘要:

Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module constructs prompt-derived surrogate rewards from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting surrogate rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑