Contemporary vision benchmarks predominantly consider tasks on which humans can achieve near-perfect performance. However, humans are frequently presented with visual data that they cannot classify with 100% certainty, and models trained on standard vision benchmarks achieve low performance when evaluated on this data. To address this issue, we introduce a procedure for creating datasets of ambiguous images and use it to produce SQUID-E ("Squidy"), a collection of noisy images extracted from videos. All images are annotated with ground truth values and a test set is annotated with human uncertainty judgments. We use this dataset to characterize human uncertainty in vision tasks and evaluate existing visual event classification models. Experimental results suggest that existing vision models are not sufficiently equipped to provide meaningful outputs for ambiguous images and that datasets of this nature can be used to assess and improve such models through model training and direct evaluation of model calibration. These findings motivate large-scale ambiguous dataset creation and further research focusing on noisy visual data.
PAS-SLAM: A Visual SLAM System for Planar Ambiguous Scenes
School of Electronic Information and Electrical Engineering, Dalian University of Technology(大连理工大学电子信息与电气工程学部);Department of Automation, Tsinghua University(清华大学自动化系);School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);College of Information Science and Engineering, Northeastern University(东北大学信息科学与工程学院)
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment
Zhejiang University(浙江大学);The University of Manchester(曼彻斯特大学);University of Oxford(牛津大学);The University of Edinburgh(爱丁堡大学);Tsinghua University(清华大学);Donghai laboratory(东海实验室)
Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment
University of Amsterdam, Amsterdam, The Netherlands(阿姆斯特丹大学,荷兰阿姆斯特丹);Aarhus University, Aarhus, Denmark(哥本哈根大学,丹麦阿arhus);Amazon AGI, Seattle, USA(亚马逊人工智能实验室,美国西雅图)
Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
University of Surrey(萨里大学);ByteDance(字节跳动);Xi'an Jiaotong-Liverpool University(西交利物浦大学);Southern University of Science and Technology(南方科技大学);Izmir Katip Celebi University(伊兹密尔卡蒂普·切莱比大学)
A Novel Biologically Mechanism-Based Visual Cognition Model--Automatic Extraction of Semantics, Formation of Integrated Concepts and Re-selection Features for Ambiguity
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Institute of Applied Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Science(中国科学院数学与系统科学研究院应用数学研究所)
Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
Bake AI;University of Washington(华盛顿大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);Stanford University(斯坦福大学);University of Notre Dame(诺丁汉大学);Carnegie Mellon University(卡内基梅隆大学);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室);Western Washington University(西雅图华盛顿大学);King Abdulaziz City for Science and Technology(国王阿卜杜勒阿齐兹科技城)
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning
Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);University of California, Berkeley(加州大学伯克利分校);Boston University(波士顿大学);MIT-IBM Watson AI(麻省理工学院-IBM沃森人工智能实验室)
人类有非凡的能力进行溯因推理,并对图像的字面内容之外的内容进行假设。通过识别分散在场景中的具体视觉线索,我们几乎忍不住根据我们对世界的日常经验和知识,在文字场景之外做出可能的推断。例如,如果我们在路边看到“20英里/小时”的标志,我们可能会认为这条街位于住宅区(而不是高速公路上),即使没有房屋被拍到。机器能执行类似的视觉推理吗?我们提出了夏洛克,103 K图像的注释语料库,用于测试机器的溯因推理能力超出字面图像内容。我们采用自由观看范式:参与者首先观察并识别图像中的显著线索(例如,物体,动作),然后根据线索对场景进行合理的推断。总的来说,我们收集了363 K(线索,推理)对,形成了第一个溯因视觉推理数据集。使用我们的语料库,我们测试三个互补轴的溯因推理。我们评估模型的能力:i)从大型候选语料库中检索相关推断; ii)通过边界框定位推断的证据; iii)比较合理的推断,以匹配新收集的19 K Likert尺度判断的诊断语料库上的人类判断。虽然我们发现使用多任务目标微调CLIP-RN 50 x64的性能优于强大的基线,但模型性能和人类一致性之间存在显着的空间。数据、模型和排行榜可在http://visualabduction.com/获得
Dynamic Visual-semantic Alignment for Zero-shot Learning with Ambiguous Labels
College of Computer Science and Technology(计算机科学与技术学院);School of Computer Science(计算机科学学院);Key Laboratory of Computing Power Network and Information Security, Ministry of Education(教育部计算能力网络与信息安全重点实验室);Shandong Computer Science Center(山东计算机科学中心);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东科学院))
I Speak and You Find: Robust 3D Visual Grounding with Noisy and Ambiguous Speech Inputs
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(计算机科学与技术学院,南京航空航天大学);Urban Data Science Section, Delft University of Technology(都市数据科学部门,代尔夫特理工大学)
StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs
Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Princeton Center for Information and Technology Policy(普林斯顿信息与技术政策中心)
The Courtroom Trial of Pixels: Robust Image Manipulation Localization via Adversarial Evidence and Reinforcement Learning Judgment
School of Computer Science and Technology, Xinjiang University(新疆大学计算机科学与技术学院);Ant Group(蚂蚁集团);College of Computer Science and Electronic Engineering, Hunan University(湖南大学计算机科学与电子工程学院)