ACAR:适应复杂度的多模型集合路由
ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces
AI总结:
ACAR是一种用于多模型集合的可审计路由框架,通过自一致性方差实现任务路由,提升准确率并避免过度融合,同时揭示归因计算的挑战。
摘要:
我们提出了ACAR(适应复杂度和归因路由),一种用于在可审计条件下研究多模型编排的测量框架。ACAR利用从N=3个探测样本计算出的自一致性方差(sigma)来在单模型、双模型和三模型执行模式之间路由任务。该系统建立在TEAMLLM之上,这是一种具有不可变 artifacts 和完整决策轨迹的确定性执行子系统。我们在四个基准测试中评估了ACAR,共1,510个任务:MathArena、Reasoning Gym、LiveCodeBench 和 SuperGPQA,使用Claude Sonnet 4、GPT-4o 和 Gemini 2.0 Flash,生成超过7,550个可审计运行。结果表明,基于sigma的路由实现了55.6%的准确率,超过双模型基线的54.4%,同时在54.2%的任务上避免了完全融合。路由机制是模型无关的,不需要任何学习组件。我们还记录了负面结果。首先,检索增强减少了准确率3.4个百分点,因为中位数检索相似度仅为0.167,表明经验注入而没有语义对齐会引入噪声而非基础。其次,当模型在错误答案上达成一致(sigma等于零)时,下游融合无法恢复;这种一致但错误的失败模式是自一致性固有的,限制了可实现的准确率大约在完全融合下的八个百分点以下。第三,基于代理信号如响应相似性和熵的归因估计与地面真实值的leave-one-out值显示出弱相关性,表明实际归因需要显式反事实计算。这项工作记录了哪些假设在实践中失效,并为未来关于路由、检索和多模型归因的研究提供了可验证的基线。
英文摘要:
We present ACAR (Adaptive Complexity and Attribution Routing), a measurement framework for studying multi-model orchestration under auditable conditions. ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes. The system is implemented on top of TEAMLLM, a deterministic execution substrate with immutable artifacts and complete decision traces. We evaluate ACAR on 1,510 tasks spanning four benchmarks: MathArena, Reasoning Gym, LiveCodeBench, and SuperGPQA, using Claude Sonnet 4, GPT-4o, and Gemini 2.0 Flash, producing more than 7,550 auditable runs. Results show that sigma-based routing achieves 55.6 percent accuracy, exceeding the two-model baseline of 54.4 percent while avoiding full ensembling on 54.2 percent of tasks. The routing mechanism is model-agnostic and requires no learned components. We also document negative results. First, retrieval augmentation reduced accuracy by 3.4 percentage points, as median retrieval similarity was only 0.167, demonstrating that experience injection without semantic alignment introduces noise rather than grounding. Second, when models agree on incorrect answers (sigma equals zero), no downstream ensemble can recover; this agreement-but-wrong failure mode is intrinsic to self-consistency and bounds achievable accuracy at approximately eight percentage points below full ensembling. Third, attribution estimates based on proxy signals such as response similarity and entropy showed weak correlation with ground-truth leave-one-out values, indicating that practical attribution requires explicit counterfactual computation. This work documents which assumptions fail in practice and provides falsifiable baselines for future research on routing, retrieval, and multi-model attribution.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。