While Machine Comprehension (MC) has attracted extensive research interests in recent years, existing approaches mainly belong to the category of Machine Reading Comprehension task which mines textual inputs (paragraphs and questions) to predict the answers (choices or text spans). However, there are a lot of MC tasks that accept audio input in addition to the textual input, e.g. English listening comprehension test. In this paper, we target the problem of Audio-Oriented Multimodal Machine Comprehension, and its goal is to answer questions based on the given audio and textual information. To solve this problem, we propose a Dynamic Inter- and Intra-modality Attention (DIIA) model to effectively fuse the two modalities (audio and textual). DIIA can work as an independent component and thus be easily integrated into existing MC models. Moreover, we further develop a Multimodal Knowledge Distillation (MKD) module to enable our multimodal MC model to accurately predict the answers based only on either the text or the audio. As a result, the proposed approach can handle various tasks including: Audio-Oriented Multimodal Machine Comprehension, Machine Reading Comprehension and Machine Listening Comprehension, in a single model, making fair comparisons possible between our model and the existing unimodal MC models. Experimental results and analysis prove the effectiveness of the proposed approaches. First, the proposed DIIA boosts the baseline models by up to 21.08% in terms of accuracy; Second, under the unimodal scenarios, the MKD module allows our multimodal MC model to significantly outperform the unimodal models by up to 18.87%, which are trained and tested with only audio or textual data.
OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
The University of Manchester（曼彻斯特大学）；The University of Hong Kong（香港大学）；Huazhong University of Science and Technology（华中科技大学）；Shanghai Academy of Educational Sciences（上海市教育科学研究院）；National University of Singapore（新加坡国立大学）
Thought-Path Contrastive Learning via Premise-Oriented Data Augmentation for Logical Reading Comprehension
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, Beijing Institute of Technology（北京理工大学海量语言信息处理与云计算应用北京工程研究中心）
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
School of Data Science, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院、深圳大数据研究院）；Bytedance（字节跳动）
Audio-FLAN: An Instruction-Following Dataset for Unified Audio Understanding and Generation of Speech, Music, and Sound
The Hong Kong University of Science（香港科学与技术大学）；Inner Mongolia University（内蒙古大学）；Beihang University（北京航空航天大学）；Queen Mary University of London（伦敦玛丽女王大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；University of Surrey（萨里大学）；University of Rochester（罗切斯特大学）；Independent Researcher（独立研究者）
