As language and visual understanding by machines progresses rapidly, we are observing an increasing interest in holistic architectures that tightly interlink both modalities in a joint learning and inference process. This trend has allowed the community to progress towards more challenging and open tasks and refueled the hope at achieving the old AI dream of building machines that could pass a turing test in open domains. In order to steadily make progress towards this goal, we realize that quantifying performance becomes increasingly difficult. Therefore we ask how we can precisely define such challenges and how we can evaluate different algorithms on this open tasks? In this paper, we summarize and discuss such challenges as well as try to give answers where appropriate options are available in the literature. We exemplify some of the solutions on a recently presented dataset of question-answering task based on real-world indoor images that establishes a visual turing challenge. Finally, we argue despite the success of unique ground-truth annotation, we likely have to step away from carefully curated dataset and rather rely on 'social consensus' as the main driving force to create suitable benchmarks. Providing coverage in this inherently ambiguous output space is an emerging challenge that we face in order to make quantifiable progress in this area.
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering
Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);UC Berkeley(加州大学伯克利分校);ICSI(国际计算机科学研究所)
我们解决了一个问答任务,对现实世界的图像,是作为一个视觉图灵测试。通过结合图像表示和自然语言处理的最新进展,我们提出了Ask Your Neurons,一个可扩展的,联合训练的,端到端的公式来解决这个问题。与以前的努力相比,我们正面临着一个多模态的问题,其中的语言输出(答案)的视觉和自然语言输入(图像和问题)的条件。我们提供了额外的见解,通过分析有多少信息只包含在语言部分,我们提供了一个新的人类基线的问题。为了研究人类的共识,这是与在这个具有挑战性的任务中固有的模糊性,我们提出了两个新的指标,并收集额外的答案,扩展原来的DAQUAR数据集DAQUAR-Consensus。此外,我们还将我们的分析扩展到VQA,一个关于图像数据集的大规模问答,在那里我们研究了一些特定的设计选择,并显示了更强大的视觉模型的重要性。与此同时,我们实现了我们的模型的强大性能,仍然使用全局图像表示。最后,基于这样的分析,我们在DAQUAR上改进了我们的Ask Your Neurons,这也导致了这项具有挑战性的任务的更好性能。
Position: Reasoning After Perception Means Reasoning Without Vision
Tsinghua University(清华大学);Peking University(北京大学);National University of Singapore(新加坡国立大学);Zhongguancun Academy(中关村学院);Nanjing University(南京大学);Nankai University(南开大学)
The Visual Counter Turing Test (VCT2): A Benchmark for Evaluating AI-Generated Image Detection and the Visual AI Index (VAI)
University of South Carolina(南卡罗来纳大学);Manipal University Jaipur(贾伊普尔Manipal大学);BITS Pilani(比斯-皮尔尼大学);Xpectrum AI(Xpectrum人工智能);International Institute of Information Technology(国际信息科技研究所);Texas A&M University(德克萨斯农工大学);Carnegie Mellon University(卡内基梅隆大学);Vishwakarma Institute of Information Technology(维什瓦克arma信息科技研究所);IIIT Delhi(德里IIIT);National Institute of Technology, Silchar(锡拉char国家理工学院);Amazon AI(亚马逊人工智能);Stanford University(斯坦福大学);Amazon GenAI(亚马逊生成人工智能);Shenzhen University of Advanced Technology(深圳先进技术大学);BITS Pilani, Goa(果阿BITS Pilani大学)
The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
School of Computer Science and Technology, East China Normal University(东华师范大学计算机科学与技术学院);Meituan Inc.(美团公司);Fudan University(复旦大学);Pazhou Laboratory(Pazhou实验室)
Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge
Intuitive, Inc.(直观医疗公司);Konica Minolta, Inc.(柯尼卡美能达公司);University College London(伦敦大学学院);University of Genoa(热那亚大学);University of Burgundy(勃艮第大学);German Cancer Research Center (DKFZ)(德国癌症研究中心);National Center for Tumor Diseases (NCT)(国家肿瘤疾病中心)
VIPriors 2: Visual Inductive Priors for Data-Efficient Deep Learning Challenges
Delft University of Technology(代尔夫特理工大学);Gradiant;Bosch Security Systems B.V.(博世安防系统有限公司);Synergy Sports
第二版“VIPriors:Visual Inductive Priors for Data-Efficient深度学习“挑战赛包含五个数据受损的挑战赛,其中模型是在各种关键计算机视觉任务的少量训练样本上从头开始训练的。为了鼓励新的和创造性的想法,结合相关的归纳偏差,以提高深度学习模型的数据效率,我们禁止使用预先训练的检查点和其他迁移学习技术。在所有五个挑战中,所提供的基线都大幅优于基线,这主要归功于广泛的数据增强策略、模型集成和数据高效网络架构。
Visual Knowledge Discovery with Artificial Intelligence: Challenges and Future Directions
Central Washington University(中央华盛顿大学);Transilvania University(特兰西瓦尼亚大学);ISEL-Instituto Superior de Engenharia de Lisboa(里斯本高等工程学院);Darmstadt University of Applied Sciences(达姆施塔特应用科学大学);London South Bank University(伦敦南岸大学)