理解并利用混合专家大语言模型中上下文忠实度的专家专业化特性
Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs
AI总结：
本研究针对大语言模型上下文忠实度不足的问题，基于混合专家架构的专家专业化特性，提出Router Lens识别上下文忠实专家，并设计轻量级CEFT方法选择性微调这些专家，实验证明其性能与全量微调相当且效率更高。
摘要：
上下文忠实度对于依赖上下文场景中的可靠推理至关重要。然而，大语言模型常难以将输出锚定在给定上下文中，导致无关响应。受混合专家架构中观察到的涌现专家专业化特性启发，本研究探究特定专家是否在上下文利用方面存在专业化，为针对性优化以提升上下文忠实度提供潜在途径。为此，我们提出Router Lens方法，能准确识别上下文忠实专家。分析显示，这些专家会逐步增强对相关上下文信息的注意力，从而提升上下文锚定能力。基于此发现，我们引入上下文忠实专家微调（CEFT），这是一种轻量级优化方法，可选择性微调上下文忠实专家。在多种基准和模型上的实验表明，CEFT的性能与全量微调相当或更优，同时效率显著更高。
英文摘要：
Context faithfulness is essential for reliable reasoning in context-dependent scenarios. However, large language models often struggle to ground their outputs in the provided context, resulting in irrelevant responses. Inspired by the emergent expert specialization observed in mixture-of-experts architectures, this work investigates whether certain experts exhibit specialization in context utilization, offering a potential pathway toward targeted optimization for improved context faithfulness. To explore this, we propose Router Lens, a method that accurately identifies context-faithful experts. Our analysis reveals that these experts progressively amplify attention to relevant contextual information, thereby enhancing context grounding. Building on this insight, we introduce Context-faithful Expert Fine-Tuning (CEFT), a lightweight optimization approach that selectively fine-tunes context-faithful experts. Experiments across a wide range of benchmarks and models demonstrate that CEFT matches or surpasses the performance of full fine-tuning while being significantly more efficient.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
