Combo of Thinking and Observing for Outside-Knowledge VQA
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);ByteDance AI Lab(字节跳动人工智能实验室)
Qingyi Si, Yuchen Mo, Zheng Lin, Huishan Ji, Weiping Wang
Outside-knowledge visual question answering is a challenging task that requires both the acquisition and the use of open-ended real-world knowledge. Some existing solutions draw external knowledge into the cross-modality space which overlooks the much vaster textual knowledge in natural-language space, while others transform the image into a text that further fuses with the textual knowledge into the natural-language space and completely abandons the use of visual features. In this paper, we are inspired to constrain the cross-modality space into the same space of natural-language space which makes the visual features preserved directly, and the model still benefits from the vast knowledge in natural-language space. To this end, we propose a novel framework consisting of a multimodal encoder, a textual encoder and an answer decoder. Such structure allows us to introduce more types of knowledge including explicit and implicit multimodal and textual knowledge. Extensive experiments validate the superiority of the proposed method which outperforms the state-of-the-art by 6.17% accuracy. We also conduct comprehensive ablations of each component, and systematically study the roles of varying types of knowledge. Codes and knowledge data can be found at https://github.com/PhoebusSi/Thinking-while-Observing.
Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
Bytedance(字节跳动);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学);Stanford University(斯坦福大学);University of Michigan(密歇根大学)
ConFoThinking: Consolidated Focused Attention Driven Thinking for Visual Question Answering
Wangxuan Institute of Computer Technology, Peking University, Beijing, China(北京大学王轩计算机技术研究所);School of AI and Advanced Computing, Xi'an Jiaotong-Liverpool University, Suzhou, China(西安交通大学利物浦大学人工智能与高级计算学院)
HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision
Southern University of Science and Technology, Shenzhen, China(南方科技大学,深圳,中国);College of William and Mary, Williamsburg, VA 23185, USA(威廉玛丽学院,威廉斯堡,VA 23185,美国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
Thinking Ahead: Foresight Intelligence in MLLMs and World Model
College of Software, Nankai University, China(南开大学软件学院);The University of Hong Kong, China(香港大学);NKIARI, Shenzhen Futian, China(NKIARI,深圳福田,中国);AAIS & VCIP, Nankai University, China(AAIS与VCIP,南开大学,中国);A*STAR Institute of Advanced Intelligence and Computing, Singapore(新加坡A*STAR先进智能与计算研究所)
Seeing Red, Thinking Bad: Color Bias in Vision Language Models
National Institute of Advanced Industrial Science and Technology (AIST)(日本产业技术综合研究所);University of Tsukuba(筑波大学);University of Technology Nuremberg(纽伦堡工业大学);University of Oxford(牛津大学)
GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagery
National University of Defense Technology, China(国防科技大学);Beijing University of Posts and Telecommunications, China(北京邮电大学);University of the Chinese Academy of Sciences, China(中国科学院大学);Shanghai Jiao Tong University, China(上海交通大学);Wuhan University, China(武汉大学);Chinese Academy of Science, China(中国科学院);Tsinghua University, China(清华大学);Renmin University of China, China(中国人民大学)
CamReasoner: Reinforcing Camera Movement Understanding via Structured Spatial Reasoning
University of California, Merced(加州大学梅尔德分校);University of Queensland(昆士兰大学);Ant Group(蚂蚁集团);University of Chinese Academy of Sciences(中国科学院大学);University at Buffalo, State University of New York(纽约州立大学布法罗分校)
Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps
School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);School of Software, Northwestern Polytechnical University(西北工业大学软件学院);University of Adelaide(阿德莱德大学)
Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models
Department of Computer Science and Informatics, Emory University(计算机科学与信息学系,埃默里大学);Department of Computer Science and Department of Electrical and Computer Engineering, University of Southern California(计算机科学系和电气与计算机工程系,南加州大学);Department of Computer Science, University of Tokyo(计算机科学系,东京大学);Department of Computer Science, Johns Hopkins University(计算机科学系,约翰霍普金斯大学);Department of Biomedical Engineering, Georgia Institute of Technology and Emory University(生物医学工程系,佐治亚理工学院和埃默里大学)