在多专家编排中解耦内在重要性与涌现结构
Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration
AI总结：
提出INFORM方法，通过解耦专家交互结构、执行顺序和功能归因，揭示路由主导性不能反映功能必要性，稀疏路由专家可能具有结构关键性。
摘要：
多专家系统，其中多个大型语言模型（LLM）协作解决复杂任务，正越来越多地被用于高性能推理和生成。然而，管理专家交互和排序的编排策略仍然很大程度上不透明。我们引入了INFORM，一种可解释性分析方法，将编排视为显式、可分析的计算，从而能够解耦专家交互结构、执行顺序和功能归因。我们使用INFORM在GSM8K、HumanEval和MMLU上评估一个编排器，采用由LLaMA-3.1 8B、Qwen3 8B和DeepSeek-R1 8B中抽取的十个指令调优专家组成的同质联盟，并控制解码温度变化，以及一个涵盖1B-7B参数模型的二次异质联盟。跨任务而言，路由主导性是功能必要性的一个糟糕代理。我们揭示了关系重要性（由路由质量和交互拓扑捕获）与内在重要性（通过梯度敏感性测量）之间的分歧：频繁被选中的专家通常充当交互枢纽但影响有限，而稀疏路由的专家可能在结构上至关重要。编排行为异步涌现，专家中心化先于稳定的路由置信度，且专家排序保持非确定性。定向消融实验表明，与屏蔽频繁出现的专家相比，屏蔽内在重要的专家会导致交互结构不成比例地崩溃，这证实了INFORM揭示了超越准确率指标的功能和结构依赖性。我们的代码可在该https URL获取。
英文摘要：
Multi-expert systems, where multiple Large Language Models (LLMs) collaborate to solve complex tasks, are increasingly adopted for high-performance reasoning and generation. However, the orchestration policies governing expert interaction and sequencing remain largely opaque. We introduce INFORM, an interpretability analysis that treats orchestration as an explicit, analyzable computation, enabling the decoupling of expert interaction structure, execution order, and functional attribution. We use INFORM to evaluate an orchestrator on GSM8K, HumanEval, and MMLU using a homogeneous consortium of ten instruction-tuned experts drawn from LLaMA-3.1 8B, Qwen3 8B, and DeepSeek-R1 8B, with controlled decoding-temperature variation, and a secondary heterogeneous consortium spanning 1B-7B parameter models. Across tasks, routing dominance is a poor proxy for functional necessity. We reveal a divergence between relational importance, captured by routing mass and interaction topology, and intrinsic importance, measured via gradient sensitivity: frequently selected experts often act as interaction hubs with limited influence, while sparsely routed experts can be structurally critical. Orchestration behaviors emerge asynchronously, with expert centralization preceding stable routing confidence and expert ordering remaining non-deterministic. Targeted ablations show that masking intrinsically important experts induces disproportionate collapse in interaction structure compared to masking frequent peers, confirming that INFORM exposes functional and structural dependencies beyond accuracy metrics alone. Our code is available at https://github.com/parmanu-lcs2/inform.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
