Unifying Specialized Visual Encoders for Video Language Models
Department of Computer Science, Princeton University, Princeton, NJ, United States（普林斯顿大学计算机科学系）；Salesforce Research, Palo Alto, CA, United States（Salesforce研究）
Jihoon Chung, Tyler Zhu, Max Gonzalez Saez-Diez, Juan Carlos Niebles, Honglu Zhou, Olga Russakovsky
大型语言模型（LLMs）的最新发展通过视频大型语言模型（VideoLLMs）将复杂推理能力引入视频领域。然而，当前VideoLLM的所有视觉处理都依赖单一视觉编码器，这限制了可传递给LLM的视觉信息的数量与类型。我们提出的方法MERV（多编码器视频表示，Multi-Encoder Representation of Videos）转而利用多个冻结的视觉编码器生成视频的统一表示，为VideoLLM提供全面的专用视觉知识集合。通过对各编码器的特征进行时空对齐，我们能够应对更广泛的开放式和多项选择视频理解问题，性能优于此前的最优工作。在标准视频理解基准套件上，MERV的准确率比Video-LLaVA最高高出3.7%，同时Video-ChatGPT得分也更优。我们还将零样本Perception Test准确率的此前最优模型SeViLA的性能提升了2.2%。MERV仅引入极少的额外参数，训练速度快于同等的单编码器方法，同时可并行处理视觉任务。最后，我们提供了定性证据，证明MERV成功从其每个编码器中捕获了领域知识。我们的研究结果为利用多视觉编码器实现全面视频理解提供了有前景的方向。
英文摘要：
The recent advent of Large Language Models (LLMs) has ushered sophisticated reasoning capabilities into the realm of video through Video Large Language Models (VideoLLMs). However, VideoLLMs currently rely on a single vision encoder for all of their visual processing, which limits the amount and type of visual information that can be conveyed to the LLM. Our method, MERV, Multi-Encoder Representation of Videos, instead leverages multiple frozen visual encoders to create a unified representation of a video, providing the VideoLLM with a comprehensive set of specialized visual knowledge. Spatio-temporally aligning the features from each encoder allows us to tackle a wider range of open-ended and multiple-choice video understanding questions and outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy than Video-LLaVA across the standard suite video understanding benchmarks, while also having a better Video-ChatGPT score. We also improve upon SeViLA, the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV introduces minimal extra parameters and trains faster than equivalent single-encoder methods while parallelizing the visual processing. Finally, we provide qualitative evidence that MERV successfully captures domain knowledge from each of its encoders. Our results offer promising directions in utilizing multiple vision encoders for comprehensive video understanding.
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Microsoft（微软）
近年来，统一视觉语言框架有了很大的发展，其中大多数采用编码器-解码器架构，将图像-文本任务统一为序列到序列生成。然而，现有的视频语言（VidL）模型仍然需要在模型架构和每个任务的训练目标的任务特定的设计。在这项工作中，我们探索了一个统一的VidL框架LAVENDER，其中Masked Language Modeling（MLM）被用作所有预训练和下游任务的公共接口。这样的统一导致简化的模型架构，其中在多模态编码器之上仅需要轻量级MLM头，而不是具有更多参数的解码器。令人惊讶的是，实验结果表明，这个统一的框架实现了竞争力的性能在14个VidL基准，涵盖视频问答，文本到视频检索和视频描述。广泛的分析进一步证明了LAVENDER优于现有VidL方法的优点：（i）在多任务微调时仅使用一组参数值支持所有下游任务;（ii）对各种下游任务的Few-Shot泛化;以及（iii）对视频问答任务启用zero-shot评估。代码可在https://github.com/microsoft/LAVENDER上获得。
Large Language Models are Powerful Electronic Health Record Encoders
Berlin Institute of Health at Charité – Universitätsmedizin Berlin（柏林夏里特医学院健康研究所）；BIH Biomedical Innovation Academy, BIH Charité Digital Clinician Scientist Program（BIH生物医学创新学院，BIH夏里特数字临床科学家计划）；Deutsches Herzzentrum der Charité – Medical Heart Center of Charité and German Heart Institute Berlin（夏里特德国心脏中心 – 夏里特医学院和德国心脏研究所柏林）；Layer Health, Inc.（Layer Health公司）
Unifying Text Semantics and Graph Structures for Temporal Text-attributed Graphs with Large Language Models
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University（上海数据科学 key laboratory，计算机科学学院，复旦大学）；School of Management, Fudan University（管理学院，复旦大学）；Tencent Weixin Group（腾讯微信集团）；IFM Lab, University of California, Davis（加州大学戴维斯分校IFM实验室）
MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Artificial Intelligence Research Institute (AIRI)（人工智能研究院）；Higher School of Economics（高等经济学院）
Unifying Adversarially Robust Model Experts in Vision-Language Models
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Center for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局前沿人工智能研究中心）
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation
The Chinese University of Hong Kong（香港中文大学）；Fudan University（复旦大学）；Sensetime Research Institute（商汤研究院）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Hong Kong University of Science and Technology（香港科技大学）；Sun Yat-sen University（中山大学）
Scope: Selective Cross-modal Orchestration of Visual Perception Experts
ServiceNow；Université de Montréal（蒙特利尔大学）；École de Technologie Supérieure（高级技术学院）；University of Waterloo（滑铁卢大学）；McGill University（麦吉尔大学）；York University（约克大学）；CIFAR AI Chair（CIFAR人工智能主席）；Mila；Law Zero
Point Clouds Are Specialized Images: A Knowledge Transfer Approach for 3D Understanding
School of Electrical and Data Engineering, University of Technology Sydney（悉尼科技大学电气与数据工程学院）；School of Computer Science, University of Nottingham Ningbo（诺丁汉大学宁波校区计算机学院）；Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（香港理工大学电机及电子工程系）
