CogniRoute: 在全模态模型中学习路由社会证据
CogniRoute: Learning to Route Social Evidence in Omni-Modal Models
AI总结：
提出CogniRoute框架，通过模式引导的专家混合和路由感知强化学习，解决全模态模型在社会视频问答中证据选择不足的问题，在OmniSocialBench上准确率达59.38%。
摘要：
全模态模型可以处理视频、音频和文本，但对多种模态的统一访问并不能保证模型使用正确的证据。这一差距在社会视频问答中尤为明显，答案可能取决于手势、语调、时间线索或所说内容与视觉表达之间的不匹配。我们提出了CogniRoute，一个用于社会全模态推理的模式引导的专家混合框架。CogniRoute使用一个仅用于训练的认知模式，该模式通过跨模态关系、推理需求和时间范围对每个示例进行分解，并在监督微调期间将全局路由签名与该结构对齐。我们进一步引入了路由感知强化学习，通过奖励答案正确性、模态一致性推理和认知时间基础，联合优化令牌生成和专家分配。为了支持训练和评估，我们构建了OmniSocialBench，一个诊断性社会视频QA资源，包含118K结构化训练示例、基础推理轨迹、模式标签、时间证据跨度以及人工验证的评估分割。CogniRoute在OmniSocialBench上达到59.38%的平均准确率，比最强的专有基线提高15.33个百分点，比最强的开源全模态基线提高26.77个百分点，在需要视听协调、冲突解决和时间基础社会推理的问题上提升最大。
英文摘要：
Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38\% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
