The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
OpenGVLab（开放视觉实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）
We present the All-Seeing Project V2: a new model and dataset designed for understanding object relations in images. Specifically, we propose the All-Seeing Model V2 (ASMv2) that integrates the formulation of text generation, object localization, and relation comprehension into a relation conversation (ReC) task. Leveraging this unified task, our model excels not only in perceiving and recognizing all objects within the image but also in grasping the intricate relation graph between them, diminishing the relation hallucination often encountered by Multi-modal Large Language Models (MLLMs). To facilitate training and evaluation of MLLMs in relation understanding, we created the first high-quality ReC dataset ({AS-V2) which is aligned with the format of standard instruction tuning data. In addition, we design a new benchmark, termed Circular-based Relation Probing Evaluation (CRPE) for comprehensively evaluating the relation comprehension capabilities of MLLMs. Notably, our ASMv2 achieves an overall accuracy of 52.04 on this relation-aware benchmark, surpassing the 43.14 of LLaVA-1.5 by a large margin. We hope that our work can inspire more future research and contribute to the evolution towards artificial general intelligence. Our project is released at https://github.com/OpenGVLab/all-seeing.
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
Due to the lack of camera parameter information for in-the-wild images, existing 3D human pose and shape (HPS) estimation methods make several simplifying assumptions: weak-perspective projection, large constant focal length, and zero camera rotation. These assumptions often do not hold and we show, quantitatively and qualitatively, that they cause errors in the reconstructed 3D shape and pose. To address this, we introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image and employs this to reconstruct 3D human bodies more accurately. First, …
A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension
University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Kuaishou Technology（快手科技）；Huazhong University of Science and Technology（华中科技大学）
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
Fudan University（复旦大学）；Huazhong University of Science and Technology（华中科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）
本文提出 The All-Seeing 项目，构建了包含超 10 亿区域标注的 AS-1B 数据集，并开发了统一框架 ASM，通过开放式语言提示和位置训练，在多种视觉语言任务中实现卓越的零样本性能。
Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
Hong Kong University of Science and Technology（香港科技大学）；School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Everlyn AI；Youtu Lab, Tencent（腾讯优图实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Central Florida（中佛罗里达大学）
Seeing Through Fog: Towards Fog-Invariant Action Recognition
Beijing Institute of Technology, Beijing, China（北京理工大学，北京，中国）；Beijing Institute for General Artificial Intelligence, Beijing, China（北京通用人工智能研究院，北京，中国）；Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China（北京理工大学扬子江地区研究院，嘉兴，中国）
ORCA: Object Recognition and Comprehension for Archiving Marine Species
Hong Kong University of Science and Technology（香港科技大学）；University of Electronic Science and Technology of China（电子科学与技术大学）；Indo Ocean Foundation Project（印度洋基金会项目）
NEC-Diff: Noise-Robust Event-RAW Complementary Diffusion for Seeing Motion in Extreme Darkness
National Key Lab of Multispectral Information Intelligent Processing Technology（多谱信息智能处理技术国家级重点实验室）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models
Tsinghua University（清华大学）；Jilin University（吉林大学）；Northwest Minzu University（西北民族大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
University of Southern California（南加州大学）；University of California San Diego（加州大学圣地亚哥分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension
Institute of Deep Perception Technology, JITRI（JITRI深度感知技术研究所）；University of Liverpool（利物浦大学）；University of Southampton（南安普顿大学）；Momoni AI（Momoni人工智能公司）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）
