VEN-VL: 一种用于高效多模态理解的视觉集成MoE框架
VEN-VL: A Visual Ensemble MoE Framework for Effective and Efficient Multi-Modal Understanding
AI总结：
提出VEN-VL框架，通过先丰富后压缩的策略，利用视觉集成MoE和自适应路由增强视觉令牌的信息容量与密度，在少量压缩令牌下实现复杂视觉任务的性能与效率平衡。
摘要：
尽管近期高效方法在加速多模态理解方面取得了显著进展，但它们仍然存在明显的性能下降。这些方法强调单一视觉线索的高压缩比，并依赖基于启发式剪枝策略的粗略注意力对齐，导致视觉令牌的信息容量和密度出现瓶颈。针对这一局限，我们提出了VEN-VL，一种遵循“先丰富后压缩”原则的视觉集成MoE框架，用于高效感知。具体来说，我们首先通过统一不同视角的视觉表示来丰富信息容量，然后通过专门视觉专家中的自适应路由器逐步压缩信息以增强信息密度。此外，我们通过显式视觉监督融入原始结构的重建能力，促进关键信息的保留。实验结果表明，我们在使用少量信息压缩令牌的复杂视觉任务中具有优越性，有效弥合了性能与效率之间的差距。
英文摘要：
Despite the remarkable progress achieved by recent efficient methods in accelerating multimodal understanding, they still suffer from noticeable performance degradation. Their emphasis on the high compression ratio of a single visual clue and reliance on the heuristic pruning strategy with coarse attention alignment incurs a bottleneck on the information capacity and density of visual tokens. Addressing this limitation, we propose VEN-VL, a visual ensemble MoE framework for effective and efficient perception following the enrich then compact principle. Specifically, we first enrich the information capacity by unifying the visual representations of different perspectives, and then progressively compact it with adaptive routers in specialized visual experts to enhance the information density. Furthermore, we incorporate the reconstruction ability of vanilla structure via explicit visual supervision, facilitating crucial information preservation. Experimental results demonstrate our superiority in complex visual tasks with few information-condensed tokens, which effectively bridges the gap between performance and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
