OrdMoE：通过多模态混合专家模型中的层次专家小组排名实现偏好对齐
OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs
AI总结：
OrdMoE通过利用混合专家架构中的内在信号，实现多模态混合专家语言模型的零成本偏好对齐，无需人工标注数据即可提升模型对齐和性能。
摘要：
偏好学习最近涌现出作为多模态大语言模型（MLLMs）后训练对齐的关键策略。然而，现有方法主要依赖于外部人工标注的偏好数据，这需要耗费成本且劳动密集。在本工作中，我们提出了OrdMoE，一种新颖的偏好对齐框架，通过利用混合专家（MoE）架构中的内在信号，完全摆脱了对外部人工偏好数据的依赖。具体来说，我们观察到路由器的专家选择分数隐含地编码了响应的质量意识排名（即得分更高的专家始终生成更高质量的输出）。基于这一见解，OrdMoE通过根据每token路由分数将专家分层到排名 tiers 中，并分别激活每个 tier 来生成一系列质量递增的响应。这产生了一个零成本的自监督偏好顺序，可以直接使用标准的偏好学习目标进行优化。在多个多模态基准上的广泛实验表明，OrdMoE显著增强了多模态混合专家语言模型的对齐和整体性能，实现了有竞争力的结果，而无需任何人工标注的偏好数据。
英文摘要：
Preference learning has recently emerged as a pivotal strategy for post-training alignment of Multimodal Large Language Models (MLLMs). However, existing approaches predominantly rely on external human-annotated preference data, which is costly and labor-intensive to collect. In this work, we propose OrdMoE, a novel preference alignment framework that bypasses the reliance on external human preferences entirely by leveraging intrinsic signals within Mixture-of-Experts (MoE) architectures. Specifically, we observe that the router's expert selection scores implicitly encode a quality-aware ranking of responses (i.e. higher-scoring experts consistently generate higher-quality outputs). Building on this insight, OrdMoE constructs an internal preference hierarchy by grouping experts into ranked tiers based on their per-token routing scores and activating each tier separately to produce a sequence of responses with increasing quality. This yields a zero-cost, self-supervised preference ordering over generated responses, which can be directly optimized using standard preference learning objectives. Extensive experiments across multiple multimodal benchmarks demnstrate that OrdMoE significantly enhances both alignment and overall performance of multimodal Mixture-of-Experts LLMs, achieving competitive results without requiring any human-annotated preference data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
