MATRIX：用于交互感知视频生成的遮罩跟踪对齐
MATRIX: Mask Track Alignment for Interaction-aware Video Generation
AI总结：
MATRIX通过引入遮罩跟踪对齐方法，提升视频生成中交互感知能力，增强语义对齐与传播，减少漂移和幻觉。
摘要：
Video DiTs在视频生成方面取得进展，但仍难以建模多实例或主体-客体交互。为此，我们构建了MATRIX-11K视频数据集，包含交互感知的描述和多实例遮罩跟踪。通过该数据集，我们系统分析了视频DiTs的两个视角：通过视频到文本注意力进行语义定位，评估名词和动词标记是否捕捉实例及其关系；通过视频到视频注意力进行语义传播，评估实例绑定是否在帧间持续。我们发现这两种效应集中在少量交互主导的层中。基于此，我们引入MATRIX，一种简单有效的正则化方法，将视频DiTs特定层的注意力与MATRIX-11K数据集的多实例遮罩跟踪对齐，从而提升语义定位和传播。我们进一步提出了InterGenEval评估协议。实验表明，MATRIX在提升交互保真度和语义对齐的同时，减少了漂移和幻觉。广泛的消融实验验证了我们的设计选择。代码和权重将被发布。
英文摘要：
Video DiTs have advanced video generation, yet they still struggle to model multi-instance or subject-object interactions. This raises a key question: How do these models internally represent interactions? To answer this, we curate MATRIX-11K, a video dataset with interaction-aware captions and multi-instance mask tracks. Using this dataset, we conduct a systematic analysis that formalizes two perspectives of video DiTs: semantic grounding, via video-to-text attention, which evaluates whether noun and verb tokens capture instances and their relations; and semantic propagation, via video-to-video attention, which assesses whether instance bindings persist across frames. We find both effects concentrate in a small subset of interaction-dominant layers. Motivated by this, we introduce MATRIX, a simple and effective regularization that aligns attention in specific layers of video DiTs with multi-instance mask tracks from the MATRIX-11K dataset, enhancing both grounding and propagation. We further propose InterGenEval, an evaluation protocol for interaction-aware video generation. In experiments, MATRIX improves both interaction fidelity and semantic alignment while reducing drift and hallucination. Extensive ablations validate our design choices. Codes and weights will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
