面向帕累托最优工具集成智能体的帕累托排名策略优化
Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization
AI总结:
提出ParetoPO框架,通过超体积引导动态标量化和帕累托排名优势计算,在多目标下优化工具使用语言模型的准确性与效率权衡。
摘要:
近期工具集成语言智能体的进展显著提升了其解决复杂推理任务的能力。然而,现有对齐方法主要关注最大化任务准确率,而忽略了工具使用效率等辅助目标,这些目标对于实际部署至关重要。为解决这一差距,我们提出ParetoPO,一个两阶段多目标优化框架,用于在竞争目标下对齐使用工具的大型语言模型(LLMs)。在第一阶段,ParetoPO利用超体积引导的动态标量化,基于全局帕累托前沿进展自适应调整奖励权重。在第二阶段,它用基于帕累托排名的优势计算替代标量化学习信号,通过优势感知的信用分配促进非支配轨迹。该设计能够在多个冲突目标间实现细粒度的动作级优化。在数学推理和多跳问答任务上的实验结果表明,与静态和启发式基线相比,ParetoPO始终能发现具有更优准确率-效率权衡的策略。
英文摘要:
Recent advances in tool-integrated language agents have significantly improved their ability to solve complex reasoning tasks. However, existing alignment methods predominantly focus on maximizing task accuracy, while overlooking auxiliary objectives such as tool-use efficiency, which are essential for practical deployment. To address this gap, we introduce ParetoPO, a two-stage multi-objective optimization framework for aligning tool-using large language models (LLMs) under competing objectives. In the first stage, ParetoPO leverages hypervolume-guided dynamic scalarization to adapt reward weights based on global Pareto frontier progress. In the second stage, it replaces scalarized learning signals with Pareto-ranking-based advantage computation, promoting nondominated trajectories through dominance-aware credit assignment. This design enables fine-grained, action-level optimization across multiple conflicting objectives. Experimental results on mathematic reasoning and multi-hop QA tasks show that ParetoPO consistently discovers policies with superior accuracy-efficiency trade-offs compared to static and heuristic baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。