Spatially Aware Multimodal Transformers for TextVQA
Georgia Institute of Technology（佐治亚理工学院）；Facebook AI Research (FAIR)（Facebook人工智能研究院（FAIR））；Google（谷歌）；University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Allen Institute for AI（艾伦人工智能研究所）
Yash Kant, Dhruv Batra, Peter Anderson, Alex Schwing, Devi Parikh, Jiasen Lu, Harsh Agrawal
Textual cues are essential for everyday tasks like buying groceries and using public transport. To develop this assistive technology, we study the TextVQA task, i.e., reasoning about text in images to answer a question. Existing approaches are limited in their use of spatial relations and rely on fully-connected transformer-like architectures to implicitly learn the spatial structure of a scene. In contrast, we propose a novel spatially aware self-attention layer such that each visual entity only looks at neighboring entities defined by a spatial graph. Further, each head in our multi-head self-attention layer focuses on a different subset of relations. Our approach has two advantages: (1) each head considers local context instead of dispersing the attention amongst all visual entities; (2) we avoid learning redundant features. We show that our model improves the absolute accuracy of current state-of-the-art methods on TextVQA by 2.2% overall over an improved baseline, and 4.62% on questions that involve spatial reasoning and can be answered correctly using OCR tokens. Similarly on ST-VQA, we improve the absolute accuracy by 4.2%. We further show that spatially aware self-attention improves visual grounding.
EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
University of Science and Technology of China（中国科学技术大学）；WeChat, Tencent Inc.（腾讯微信）；Fudan University（复旦大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kuaishou Technology（快手科技）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人创新中心）
FlexAttention for Efficient High-Resolution Vision-Language Models
UMass Amherst（马萨诸塞大学阿默斯特分校）；Princeton University（普林斯顿大学）；South China University of Technology（华南理工大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing
Institute of Remote Sensing and Geographic Information System, School of Earth and Space Sciences, Peking University（遥感与地理信息系统研究所，地球与空间科学学院，北京大学）；Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology（土木与环境工程系，香港科学与技术大学）
MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models
School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University, Guangzhou 510006, China（计算机科学与工程学院，信息技术MOE实验室，广东省信息安全技术重点实验室，中山大学，广州510006，中国）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；Department of Computer and Information Science, University of Macau.（计算机与信息科学系，澳门大学）
Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruning
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究学院）
Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd（星辰通用人工智能实验室，中国电信人工智能技术（北京）有限公司）；University of Science and Technology Beijing（北京科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）
Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
School of Computer Science, Beijing Institute of Technology（北京理工大学计算机科学学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Xiaohongshu Inc（小红书公司）
Stitch and Tell通过结构化空间监督提升视觉-语言模型的空间理解能力，有效缓解空间幻觉并提高相关任务性能。
Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detection
Fusing Pixels and Genes: Spatially-Aware Learning in Computational Pathology
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Fysics Intelligence Technologies Co., Ltd. (Fysics AI)（菲茨斯智能科技有限公司（菲茨斯AI））；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）；Tencent Youtu Lab（腾讯优图实验室）；ByteDance（字节跳动）
Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding
Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Sensetime
UrbanGraphEmbeddings: Learning and Evaluating Spatially Grounded Multimodal Embeddings for Urban Science
National University of Singapore Department of Architecture Singapore；The Chinese University of Hong Kong Dept of Systems Eng. \& Eng. Mgmt. China；Singapore Management University School of Computing \& Info. Systems Singapore；National University of Singapore；Department of Architecture；The Chinese University of Hong Kong；Singapore Management University
InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Zhengzhou Advanced Research Institute of Harbin Institute of Technology（哈尔滨工业大学郑州先进研究院）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术大学人工智能研究院）
