The current deep neural network algorithm still stays in the end-to-end training supervision method like Image-Label pairs, which makes traditional algorithm is difficult to explain the reason for the results, and the prediction logic is difficult to understand and analyze. The current algorithm does not use the existing human knowledge information, which makes the model not in line with the human cognition model and makes the model not suitable for human use. In order to solve the above problems, the present invention provides a deep neural network training method based on the human knowledge, which uses the human cognition model to construct the deep neural network training model, and uses the existing human knowledge information to construct the deep neural network training model. This paper proposes a multi-level hierarchical deep learning algorithm, which is composed of multi-level hierarchical deep neural network architecture and multi-level hierarchical deep learning framework. The experimental results show that the proposed algorithm can effectively explain the hidden information of the neural network. The goal of our study is to improve the interpretability of deep neural networks (DNNs) by providing an analysis of the impact of knowledge injection on the classification task. We constructed a knowledge injection dataset with matching knowledge data and image classification data. The knowledge injection dataset is the benchmark dataset for the experiments in the paper. Our model expresses the improvement in interpretability and classification task performance of hidden layers at different scales.
Zhejiang University（浙江大学）；University of Oxford（牛津大学）；The University of Edinburgh（爱丁堡大学）
Visual question answering (VQA) often requires an understanding of visual concepts and language semantics, which relies on external knowledge. Most existing methods exploit pre-trained language models or/and unstructured text, but the knowledge in these resources are often incomplete and noisy. Some other methods prefer to use knowledge graphs (KGs) which often have intensive structured knowledge, but the research is still quite preliminary. In this paper, we propose LaKo, a knowledge-driven VQA method via Late Knowledge-to-text Injection. To effectively incorporate an external KG, we transfer…
The University of Hong Kong（香港大学）；University of Cambridge（剑桥大学）；Shenzhen Bay Laboratory（深圳湾实验室）
Whole slide image (WSI) has been widely used to assist automated diagnosis under the deep learning fields. However, most previous works only discuss the SINGLE task setting which is not aligned with real clinical setting, where pathologists often conduct multiple diagnosis tasks simultaneously. Also, it is commonly recognized that the multi-task learning paradigm can improve learning efficiency by exploiting commonalities and differences across multiple tasks. To this end, we present a novel multi-task framework (i.e., MulGT) for WSI analysis by the specially designed Graph-Transformer equippe…
Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge
Beijing Institute of Technology, Beijing, China（北京理工大学）；National University of Singapore, Singapore（新加坡国立大学）；Tsinghua University, Beijing, China（清华大学）；The Hong Kong Polytechnic University, Hong Kong（香港理工大学）
Knowledge-Based Visual Question Answering (KBVQA) is a bi-modal task requiring external world knowledge in order to correctly answer a text question and associated image. Recent single modality text work has shown knowledge injection into pre-trained language models, specifically entity enhanced knowledge graph embeddings, can improve performance on downstream entity-centric tasks. In this work, we empirically study how and whether such methods, applied in a bi-modal setting, can improve an existing VQA system's performance on the KBVQA task. We experiment with two large publicly available VQA…
External Knowledge Injection for CLIP-Based Class-Incremental Learning
School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）
Enhancing Chest X-ray Classification through Knowledge Injection in Cross-Modality Learning
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Westlake University Research Center for Industries of the Future（西湖大学未来产业研究中心）；The Second Affiliated Hospital Zhejiang University School of Medicine (SAHZU)（浙江大学医学院附属第二医院）；Google Developer Relations Team（谷歌开发者关系团队）
RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injection
Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学可信自主系统研究院和计算机科学与工程系）；School of Computer Science, University of Nottingham Ningbo China（宁波大学计算机学院）
Few-Shot Generative Model Adaption via Identity Injection and Preservation
School of Cyberspace Security, Hangzhou Dianzi University（杭州电子科技大学网络安全学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；Lishui University and Lishui Institute of Hangzhou Dianzi University（丽水大学和杭州电子科技大学丽水研究所）；School of Faculty of Applied Science, Macao Polytechnic University（澳门理工学院应用科学学院）；School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）
Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model
University of Science and Technology Beijing（北京科技大学）；Hefei University of Technology（合肥工业大学）；University of Auckland（奥克兰大学）；Nanjing University of Science and Technology（南京理工大学）；Swansea University（斯旺西大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
