AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
ServiceNow;York University(约克大学);Mila – Quebec AI Institute(魁北克人工智能研究院);École de Technologie Supérieure(魁北克高等技术学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);University of Waterloo(滑铁卢大学);CIFAR AI Chair(CIFAR人工智能 chair);Polytechnique Montréal(蒙特利尔理工学院);University of British Columbia(不列颠哥伦比亚大学)
Ahmed Masry, Juan A. Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang, Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-André Noël, Sathw… 展开作者
Ahmed Masry, Juan A. Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang, Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-André Noël, Sathwik Tejaswi Madhusudhan, Marco Pedersoli, Bang Liu, Nicolas Chapados, Yoshua Bengio, Enamul Hoque, Christopher Pal, Issam H. Laradji, David Vazquez, Perouz Taslakian, Spandana Gella, Sai Rajeswar
Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), lack inductive bias to constrain visual features within the linguistic structure of the LLM's embedding space, making them data-hungry and prone to cross-modal misalignment. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where visual and textual modalities are highly correlated. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods, with larger gains on document understanding tasks and under low-resource setups. We provide further analysis demonstrating its efficiency and robustness to noise.
Interpretable probabilistic embeddings: bridging the gap between topic models and neural networks
National Research University Higher School of Economics(国立研究大学高等经济学院);Lomonosov Moscow State University(莫斯科国立大学);Moscow Institute of Physics and Technology(莫斯科物理技术学院)
Bridging the Question-Answer Gap in Retrieval-Augmented Generation: Hypothetical Prompt Embeddings
University of Primorska(普里莫尔斯卡大学);InnoRenew CoE;Research Centre of the Slovenian Academy of Sciences and Arts(斯洛文尼亚科学与艺术研究院研究中心);The Fran Ramovš Institute(弗兰·拉莫夫什研究所)
Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications
University of Connecticut(康涅狄格大学);University of North Carolina at Charlotte(北卡罗来纳大学夏洛特分校);University of California, Riverside(加州大学河滨分校);University of Pittsburgh(匹兹堡大学);Carnegie Mellon University(卡内基梅隆大学);San Francisco State University(旧金山州立大学)
Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Alibaba Cloud Computing(阿里云计算);Hong Kong University of Science and Technology(香港科技大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校)
Bridging the Knowledge Void: Inference-time Acquisition of Unfamiliar Programming Languages for Coding Tasks
State Key Laboratory for Novel Software Technology, Nanjing University, China(新型软件技术国家重点实验室,南京大学,中国);National Institute of Healthcare Data Science, Nanjing University, China(健康数据科学国家研究院,南京大学,中国)