Understanding spatial relations is essential for intelligent agents to act and communicate in the physical world. Relative directions are spatial relations that describe the relative positions of target objects with regard to the intrinsic orientation of reference objects. Grounding relative directions is more difficult than grounding absolute directions because it not only requires a model to detect objects in the image and to identify spatial relation based on this information, but it also needs to recognize the orientation of objects and integrate this information into the reasoning process. We investigate the challenging problem of grounding relative directions with end-to-end neural networks. To this end, we provide GRiD-3D, a novel dataset that features relative directions and complements existing visual question answering (VQA) datasets, such as CLEVR, that involve only absolute directions. We also provide baselines for the dataset with two established end-to-end VQA models. Experimental evaluations show that answering questions on relative directions is feasible when questions in the dataset simulate the necessary subtasks for grounding relative directions. We discover that those subtasks are learned in an order that reflects the steps of an intuitive pipeline for processing relative directions.
Focus on What Really Matters in Low-Altitude Governance: A Management-Centric Multi-Modal Benchmark with Implicitly Coordinated Vision-Language Reasoning Framework
National University of Defense Technology(国防科技大学);Zidong Taichu (Beijing) Technology Co., Ltd.(紫东太初(北京)科技有限公司);Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Wuhan AI Research(武汉人工智能研究院)
From What to Which: Decoding Modifier Grounding in Frozen MLLMs
AI for Good (AIGO)(人工智能促进美好(AIGO));Istituto Italiano di Tecnologia(意大利技术研究院);University of Siena(锡耶纳大学);University of Genoa(热那亚大学);University of Verona(维罗纳大学)
What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions
Columbia University(哥伦比亚大学);Goethe University Frankfurt(法兰克福大学);University of Bonn(波恩大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Quality Match GmbH(Quality Match有限公司);IBM Research AI(IBM研究院人工智能部门);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs
Guangming Laboratory(光明实验室);National University of Singapore(新加坡国立大学);Peking University(北京大学);University of Waterloo(滑铁卢大学);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳))
Portland State University(波特兰州立大学);Stanford University(斯坦福大学);Santa Fe Institute(圣塔菲研究所)
我们重新审视了一种特定的视觉基础方法:Johnson et al.(2015)的“Image Retrieval Using Scene Graphs”(IRSG)系统。我们的实验表明,该系统没有有效地使用其学习的对象关系模型。我们还仔细研究了IRSG数据集,以及广泛使用的视觉关系数据集(VRD),我们发现这些数据集表现出偏差,允许忽略关系的方法相对较好地执行。我们还描述了IRSG数据集的其他几个问题,并报告了使用数据集子集的实验,其中消除了偏差和其他问题。我们的研究有助于更广泛的努力:更好地理解结合语言和视觉的机器学习方法实际上学习了什么,以及流行的数据集实际上测试了什么。
Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback
Tsinghua University(清华大学);Kolors Team, Kuaishou Technology(快手科技Kolors团队);University of British Columbia(不列颠哥伦比亚大学);Vector Institute(向量研究所);South China Normal University(华南师范大学)
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
University of Technology Sydney(悉尼科技大学);East China Normal University(华东师范大学);The University of New South Wales(新南威尔士大学);Shanghai Jiaotong University(上海交通大学);The Education University of Hong Kong(香港教育大学)
ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision
IBM Research Zurich, Zurich, Switzerland(IBM苏黎世研究实验室,瑞士苏黎世);ETH Zurich, Computer Vision(苏黎世联邦理工学院,计算机视觉);ETH AI Center, Switzerland(苏黎世联邦理工学院人工智能中心,瑞士);ETH Zurich, Photogrammetry(苏黎世联邦理工学院,摄影测量学);Microsoft, Switzerland(微软公司,瑞士)
GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed
XPeng Inc.(小鹏汽车);Peking University(北京大学);The University of Hong Kong(香港大学);University of California, Berkeley(加州大学伯克利分校);Princeton University(普林斯顿大学);National University of Singapore(新加坡国立大学);HKUST (GZ)(香港科技大学(广州))
LLM-RG: Referential Grounding in Outdoor Scenarios using Large Language Models
Birla Institute of Technology and Science Pilani, K.K Birla Goa Campus(比拉理工学院和科学学院,K.K Birla Goa校区);Carnegie Mellon University, Robotics Institute(卡内基梅隆大学,机器人研究所)
What's left can't be right -- The remaining positional incompetence of contrastive vision-language models
SFI Centre for Research Training in Machine Learning at University College Dublin(都柏林大学学院SFI机器学习研究培训中心);School of Computer Science, University College Dublin(都柏林大学学院计算机科学学院);School of Computer Science and Statistics, Trinity College Dublin(都柏林圣三一学院计算机科学与统计学院)