DRDF：用双路由器动态框架确定不同多模态信息的重要性
DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
摘要：
在多模态任务中，我们发现文本和图像模态信息的重要性对于不同的输入情况是不同的，并为此动机，我们提出了一个高性能和高度通用的双路由器动态框架（DRDF），由双路由器，MWF-Layer，专家和专家融合单元。双路由器中的文本路由器和图像路由器接受文本模态信息和图像模态信息，并使用MWF-Layer来确定模态信息的重要性。MWF-Layer根据判定结果生成融合权重，用于专家融合。专家是匹配当前任务的模型骨干。DRDF具有高性能和高通用性，我们已经在多模态数据集Hateful memes，单峰数据集CIFAR 10，CIFAR 100和TinyImagenet上测试了12个主干，如Visual BERT。我们的DRDF优于所有基准。通过烧蚀实验对DRDF的组成进行了详细的验证，并对DRDF设计的原因和思路进行了比较和讨论。
英文摘要：
In multimodal tasks, we find that the importance of text and image modal information is different for different input cases, and for this motivation, we propose a high-performance and highly general Dual-Router Dynamic Framework (DRDF), consisting of Dual-Router, MWF-Layer, experts and expert fusion unit. The text router and image router in Dual-Router accept text modal information and image modal information, and use MWF-Layer to determine the importance of modal information. Based on the result of the determination, MWF-Layer generates fused weights for the fusion of experts. Experts are model backbones that match the current task. DRDF has high performance and high generality, and we have tested 12 backbones such as Visual BERT on multimodal dataset Hateful memes, unimodal dataset CIFAR10, CIFAR100, and TinyImagenet. Our DRDF outperforms all the baselines. We also verified the components of DRDF in detail by ablations, compared and discussed the reasons and ideas of DRDF design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
