Augmented and Mixed Reality are emerging as likely successors to the mobile internet. However, many technical challenges remain. One of the key requirements of these systems is the ability to create a continuity between physical and virtual worlds, with the user's visual perception as the primary interface medium. Building this continuity requires the system to develop a visual understanding of the physical world. While there has been significant recent progress in computer vision and AI techniques such as image classification and object detection, success in these areas has not yet led to the visual perception required for these critical MR and AR applications. A significant issue is that current evaluation criteria are insufficient for these applications. To motivate and evaluate progress in this emerging area, there is a need for new metrics. In this paper we outline limitations of current evaluation criteria and propose new criteria.
ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving
Hong Kong University of Science and Technology（香港科学与技术大学）；The Chinese University of Hong Kong（香港中文大学）；Dalian University of Technology（大连理工大学）；TU Delft（代尔夫特理工大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Huawei Zurich Research Center（华为苏黎世研究中心）；Huawei IAS BU（华为IAS业务单元）
EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery
School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；Wenzhou Medical University（温州医科大学）；School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）
HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Alibaba（阿里巴巴）；The Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）
ScanFormer: Referring Expression Comprehension by Iteratively Scanning
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang-Singapore Innovation and Al Joint Research Lab（浙新创新与人工智能联合研究实验室）
Evaluating Graphical Perception Capabilities of Vision Transformers
Viscom Group, Institute of Media Informatics, Ulm University（Viscom集团，媒体信息学研究所，乌尔姆大学）；ViRVIG Group, Universitat Politècnica de Catalunya - BarcelonaTech（ViRVIG集团，巴塞罗那技术大学-加泰罗尼亚理工大学）；Ulm University（乌尔姆大学）
MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension
College of Systems Engineering, National University of Defense Technology（系统工程学院，国防科技大学）；Tsinghua University（清华大学）；Hefei University of Technology（合肥工业大学）；iFLYTEK Research（iFLYTEK研究院）
Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception
University of Limerick（利默里克大学）；Data Driven Computer Engineering Research Centre（数据驱动计算机工程研究中心）；Lero, The Irish Software Research Centre（Lero爱尔兰软件研究中心）；Valeo Vision Systems（瓦莱欧视觉系统）
Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
AIML, University of Adelaide（AIML，阿德莱德大学）；Monash University（墨尔本大学）；Australian National University（澳大利亚国立大学）；Tiktok, Australia（Tiktok，澳大利亚）；Zhejiang University（浙江大学）
Occluded nuScenes: A Multi-Sensor Dataset for Evaluating Perception Robustness in Automated Driving
University of Limerick（利默里克大学）；Data Driven Computer Engineering Research Centre（数据驱动计算机工程研究中心）；Lero, The Irish Software Research Centre（Lero爱尔兰软件研究中心）；Valeo Vision Systems（Valéo视觉系统）；Queen Mary University of London（伦敦女王学院）
Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs
Institute of Computing Technology, Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；China University of Geosciences（中国地质大学）
Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension
Institute of Deep Perception Technology, JITRI（JITRI深度感知技术研究所）；University of Liverpool（利物浦大学）；University of Southampton（南安普顿大学）；Momoni AI（Momoni人工智能公司）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）
Dual Enhancement on 3D Vision-Language Perception for Monocular 3D Visual Grounding
School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；College of Information Science and Engineering, Hunan Normal University（信息科学与工程学院，湖南师范大学）；School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）
The University of Hong Kong（香港大学）；University of Wollongong（伍伦贡大学）；Tencent AI Lab（腾讯人工智能实验室）；Australian Centre for Robotic Vision, University of Adelaide（阿德莱德大学澳大利亚机器人视觉中心）
Referring expression comprehension (REF) aims at identifying a particular object in a scene by a natural language expression. It requires joint reasoning over the textual and visual domains to solve the problem. Some popular referring expression datasets, however, fail to provide an ideal test bed for evaluating the reasoning ability of the models, mainly because 1) their expressions typically describe only some simple distinctive properties of the object and 2) their images contain limited distracting information. To bridge the gap, we propose a new dataset for visual reasoning in context of …
Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It thus more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In…
Huawei Munich Research Center（华为慕尼黑研究中心）；Univ. Autònoma de Barcelona (UAB)（巴塞罗那自治大学）；Computer Vision Center (CVC)（计算机视觉中心）
Monocular Depth Estimation (MDE) is performed to produce 3D information that can be used in downstream tasks such as those related to on-board perception for Autonomous Vehicles (AVs) or driver assistance. Therefore, a relevant arising question is whether the standard metrics for MDE assessment are a good indicator of the accuracy of future MDE-based driving-related perception tasks. We address this question in this paper. In particular, we take the task of 3D object detection on point clouds as a proxy of on-board perception. We train and test state-of-the-art 3D object detectors using 3D poi…
EAGLE: Elevating Geometric Reasoning through LLM-empowered Visual Instruction Tuning
Department of Computer Science, Western University（计算机科学系，西部大学）；Meituan Inc.（美团公司）；Department of Automation, Tsinghua University（自动化系，清华大学）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院，东华大学）
WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models
Tsinghua University, Beijing, China（清华大学）；Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；The University of Hong Kong, Hong Kong SAR, China（香港大学）；Princeton University, Princeton, NJ, USA（普林斯顿大学）；Chinese Academy of Sciences, Beijing, China（中国科学院）；University of Science（科学技术大学）；Peking University, Beijing, China（北京大学）；National University of Singapore, Singapore（新加坡国立大学）
