Scene Graph Generation (SGG) encodes visual relationships between objects in images as graph structures. Thanks to the advances of Vision-Language Models (VLMs), the task of Open-Vocabulary SGG has been recently proposed where models are evaluated on their functionality to learn a wide and diverse range of relations. Current benchmarks in SGG, however, possess a very limited vocabulary, making the evaluation of open-source models inefficient. In this paper, we propose a new reference-free metric to fairly evaluate the open-vocabulary capabilities of VLMs for relation prediction. Another limitation of Open-Vocabulary SGG is the reliance on weakly supervised data of poor quality for pre-training. We also propose a new solution for quickly generating high-quality synthetic data through region-specific prompt tuning of VLMs. Experimental results show that pre-training with this new data split can benefit the generalization capabilities of Open-Voc SGG models.
Google Research（谷歌研究院）；Cornell University（康奈尔大学）；University of Copenhagen（哥本哈根大学）
Utilizing vision and language models (VLMs) pre-trained on large-scale image-text pairs is becoming a promising paradigm for open-vocabulary visual recognition. In this work, we extend this paradigm by leveraging motion and audio that naturally exist in video. We present \textbf{MOV}, a simple yet effective method for \textbf{M}ultimodal \textbf{O}pen-\textbf{V}ocabulary video classification. In MOV, we directly use the vision encoder from pre-trained VLMs with minimal modifications to encode video, optical flow and audio spectrogram. We design a cross-modal fusion mechanism to aggregate compl…
Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone
Guru Gobind Singh Indraprastha University（古鲁·戈宾德·辛格印度教普拉斯塔大学）；AI Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Indian Institute of Technology Patna, India（印度理工学院帕纳布分校）
SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoning
The Laboratory of Intelligent Collaborative Computing of UESTC（UESTC智能协同计算实验室）；Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province（四川省 Ubiquitous Intelligence and Trusted Services 重点实验室）；Monash University（墨尔本大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））
Open-Vocabulary Domain Generalization in Urban-Scene Segmentation
Department of Information Engineering and Computer Science, University of Trento（信息工程与计算机科学系，特伦托大学）；School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院，合肥工业大学）
MROVSeg: Breaking the Resolution Curse of Vision-Language Models in Open-Vocabulary Image Segmentation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院 通用人工智能国家重点实验室）；School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）
ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
Artificial Intelligence and Intelligent Information Systems, University of Trier（人工智能与智能信息系统，特里尔大学）；German Research Center for Artificial Intelligence (DFKI), Trier（德国人工智能研究中心（DFKI），特里尔）
Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration
Wangxuan Institute of Computer Technology, Peking University（北京大学王璇计算机技术研究所）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）
PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation
Nanjing University of Science and Technology（南京理工大学）；Nanjing Normal University（南京师范大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（成均馆大学电子与计算机工程系）；University of Electronic Science and Technology of China（电子科技大学）；State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery（先进施工机械智能制造国家重点实验室）
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable sc…
Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；UAS Esslingen（埃森嫩大学）；TU Wien（维也纳技术大学）；Dr. Ing. h.c. F. Porsche AG（保时捷股份有限公司）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Interdisciplinary Institute of Artificial Intelligence, Xidian University（西安电子科技大学跨学科人工智能研究院）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；Northwest Institute of Nuclear Technology（西北核技术研究所）；School of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）
ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
Massachusetts Institute of Technology（麻省理工学院）；American International School Vienna（维也纳美国国际学校）；Substrate Labs；University of North Florida（北佛罗里达大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Motork；Collingwood School（科林伍德学校）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）
VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models
University of Southern California（南加州大学）；University of Bristol（布里斯托大学）；Centre National de la Recherche Scientifique（法国国家科学研究中心）；University of California, Los Angeles（加州大学洛杉矶分校）
