The interest in Artificial Intelligence (AI) and its applications has seen unprecedented growth in the last few years. The success can be partly attributed to the advancements of deep neural networks made in the sub-fields of AI such as Computer Vision (CV) and Natural Language Processing (NLP). The promising research area that this dissertation focuses on is visual and language understanding which involves many challenging tasks, i.e., classification, detection, segmentation, machine translation and captioning, etc. The state-of-the-art methods for solving these problems usually involves only two parts: source data and target labels, which is rather insufficient especially when the dataset is small. Meanwhile, many external tools or sources can provide extra useful information (external knowledge) that can help improve the performance of these methods. For example, a detection model has been applied to provide better object features than state-of-the-art ResNet for image captioning models. Inspired by this observation, we developed a methodology that we can first extract external knowledge and then integrate it with the original models. The external knowledge has to be extracted from the dataset, or can directly come from external, e.g., grammar rules or scene graphs. We apply this methodology to different AI tasks, including machine translation and image captioning and improve the original state-of-the-art models by a large margin.
Exploring Diverse Methods in Visual Question Answering
University of Michigan（密歇根大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Chongqing University（重庆大学）；Columbia University（哥伦比亚大学）；University of Chicago（芝加哥大学）
Things not Written in Text: Exploring Spatial Commonsense from Visual Signals
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；The MOE Key Laboratory of Computational Linguistics, Peking University（北京大学计算语言学教育部重点实验室）；Artificial Intelligence Institute of Peking University（北京大学人工智能研究院）；State Key Laboratory of Media Convergence Production Technology and Systems（媒体融合生产技术与系统国家重点实验室）
Exploring Image Representation with Decoupled Classical Visual Descriptors
The MIx Group University of Birmingham, UK（米克斯小组英国伯明翰大学）；University of Cambridge Cambridge, UK（剑桥大学）；Allsee Technologies Ltd Birmingham, UK（Allsee技术有限公司）
VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））
Exploring adaptation of VideoMAE for Audio-Visual Diarization & Social @ Ego4d Looking at me Challenge
Beijing University of Posts and Telecommunications（北京邮电大学）；Nanjing University（南京大学）
在本报告中，我们介绍了将预训练的视频掩码自动编码器（VideoMAE）转移到Ego 4d Looking at me Challenge的自我中心任务。VideoMAE是用于自我监督视频预训练的数据高效预训练模型，可以轻松地转移到下游任务。我们表明，从VideoMAE传输的表示具有良好的时空建模和捕捉小动作的能力。我们只需要使用以自我为中心的数据来训练基于VideoMAE的10个epoch，其中VideoMAE是由从第三人的视角获取的普通视频进行预训练的，并且我们可以得到比Ego 4d Looking at me Challenge的基线更好的结果。
Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
Fudan University（复旦大学）；Alibaba Group Holding Limited（阿里巴巴集团控股有限公司）；Future Living Lab of Alibaba（阿里巴巴未来生活实验室）；University of Southern California（南加州大学）；Shanghai Innovation Institute（上海创新研究院）
Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Microsoft Research Asia（微软亚洲研究院）；School of Telecommunication and Engineering, Xidian University（西安电子科技大学电信与工程学院）
Exploring Cluster Analysis in Nelore Cattle Visual Score Attribution
Universidade Católica Dom Bosco（多姆博斯科天主教大学）；Kerow Soluções de Precisao（凯罗精准解决方案公司）；Universidade Estadual de Mato Grosso do Sul（南马托格罗索州立大学）；Embrapa Beef Cattle（巴西农牧业研究院肉牛研究中心）；Universidade Federal de Mato Grosso do Sul（南马托格罗索联邦大学）
Exploring Visual Interpretability for Contrastive Language-Image Pre-training
The Hong Kong University of Science and Technology（香港科技大学）；University of Technology Sydney（悉尼科技大学）
对比图像预训练（CLIP）通过对自然语言的现成监督来学习丰富的表示。它提高了下游视觉任务的性能，包括但不限于zero-shot、长尾、分割、检索、字幕和视频。然而，CLIP的视觉可解释性研究很少，特别是对于原始特征图。为了提供其预测的视觉解释，我们提出了图像-文本相似性图（ITSM）。在此基础上，我们惊讶地发现，CLIP喜欢的背景区域比前景，并显示错误的可视化结果对人类的理解。这种现象在视觉Transformer和卷积网络中都是普遍存在的，这表明这个问题是独特的，而不是由于特定的网络。通过实验，我们发现问题出在池化部分，不适当的池化方法会导致一种称为语义转换的现象。针对这个问题，我们提出了可解释的对比图像预训练（ECLIP），它通过Masked Max Pooling来纠正可解释性。具体来说，为了避免语义转移，我们用max pooling代替了原来的注意力池，专注于自信的前景，并在训练过程中得到自由注意力的指导。在三个数据集上的实验表明，ECLIP大大提高了CLIP的可解释性，并在很大程度上超过了以前的可解释性方法。代码将在稍后发布。
PSVMA+: Exploring Multi-granularity Semantic-visual Adaption for Generalized Zero-shot Learning
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）
Exploring Visual Context for Weakly Supervised Person Search
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China（上海交通大学人工智能研究院人工智能教育部重点实验室）；Inception Institute of Artificial Intelligence (IIAI), UAE（阿联酋人工智能初创研究所(Inception Institute of Artificial Intelligence, IIAI)）
Exploring Long Tail Visual Relationship Recognition with Large Vocabulary
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；IIT Roorkee（印度理工学院鲁尔基分校）；Stanford University（斯坦福大学）；Baidu（百度）；NTU Singapore（新加坡南洋理工大学）
