Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models
The University of Tokyo（东京大学）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；Duke University（杜克大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；LY Corporation（LY公司）；Tokyo University of Science（东京科学大学）
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, Kiyoharu Aizawa
本文提出了一项用于评估大型多模态模型（LMM）鲁棒理解能力的新任务，称为“不可解问题检测（Unsolvable Problem Detection，UPD）”。多项选择问答（MCQA）被广泛用于评估LMM的理解能力，但它并不能保证LMM真正理解答案。UPD评估LMM在遇到MCQA中的不可解问题时弃权（不执行）作答的能力，从而验证模型是否真正理解答案。UPD包含三类问题：缺失答案检测（Absent Answer Detection，AAD）、不相容答案集检测（Incompatible Answer Set Detection，IASD）和不相容视觉问题检测（Incompatible Visual Question Detection，IVQD），涵盖答案缺失或选项不相容、图像与问题不匹配等不可解情形。为开展评估，作者提出MM-UPD Bench，这是一个用于评估多种能力维度表现的基准。实验表明，即便是大多数在现有基准上表现充分的LMM，也在MM-UPD上明显遇挫，这凸显了当前基准所忽视的一个可信性新维度。详细分析显示，LMM存在不同瓶颈；对于瓶颈位于LLM能力的LMM，思维链和自我反思能够提升性能。作者希望这些洞见能够增进更广泛的理解，并推动更可靠LMM的开发。代码位于https://github.com/AtsuMiyai/UPD。
英文摘要：
This paper introduces a novel task to evaluate the robust understanding capability of Large Multimodal Models (LMMs), termed $\textbf{Unsolvable Problem Detection (UPD)}$. Multiple-choice question answering (MCQA) is widely used to assess the understanding capability of LMMs, but it does not guarantee that LMMs truly comprehend the answer. UPD assesses the LMM's ability to withhold answers when encountering unsolvable problems of MCQA, verifying whether the model truly understands the answer. UPD encompasses three problems: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Visual Question Detection (IVQD), covering unsolvable cases like answer-lacking or incompatible choices and image-question mismatches. For the evaluation, we introduce the MM-UPD Bench, a benchmark for assessing performance across various ability dimensions. Our experiments reveal that even most LMMs, which demonstrate adequate performance on existing benchmarks, struggle significantly with MM-UPD, underscoring a novel aspect of trustworthiness that current benchmarks have overlooked. A detailed analysis shows that LMMs have different bottlenecks and chain-of-thought and self-reflection improved performance for LMMs with the bottleneck in their LLM capability. We hope our insights will enhance the broader understanding and development of more reliable LMMs. The code is available at https://github.com/AtsuMiyai/UPD.
We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answ…
Lecture slide presentations, a sequence of pages that contain text and figures accompanied by speech, are constructed and presented carefully in order to optimally transfer knowledge to students. Previous studies in multimedia and psychology attribute the effectiveness of lecture presentations to their multimodal nature. As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational cont…
Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent research trend in natural language processing that models natural language as it happens in face-to-face communication. Although humor detection is an established research area in NLP, in a multimodal context it is an understudied area. This paper presents a diverse multimodal dat…
Recent efforts of multimodal Transformers have improved Visually Rich Document Understanding (VrDU) tasks via incorporating visual and textual information. However, existing approaches mainly focus on fine-grained elements such as words and document image patches, making it hard for them to learn from coarse-grained elements, including natural lexical units like phrases and salient visual regions like prominent image regions. In this paper, we attach more importance to coarse-grained elements containing high-density information and consistent semantics, which are valuable for document understa…
SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding
Hong Kong Institute of Science & Innovation（香港科学与工业创新研究院）；CAMP, Technische Universität München（CAMP，慕尼黑技术大学）；Department of Surgery, Faculty of Medicine, The Chinese University of Hong Kong（香港中文大学医学院外科部）
ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
Nankai University（南开大学）；Shanghai Innovation Institute（上海创新研究院）；University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；Shanghai AI Laboratory（上海人工智能实验室）
Decoupled Vision-Language System for Multimodal Understanding and Generation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Li Auto（理想汽车）
In multimodal assistant, where vision is also one of the input modalities, the identification of user intent becomes a challenging task as visual input can influence the outcome. Current digital assistants take spoken input and try to determine the user intent from conversational or device context. So, a dataset, which includes visual input (i.e. images or videos for the corresponding questions targeted for multimodal assistant use cases, is not readily available. The research in visual question answering (VQA) and visual question generation (VQG) is a great step forward. However, they do not …
Docopilot: Improving Multimodal Models for Document-Level Understanding
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）；Nankai University（南开大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）
EHWGesture -- A dataset for multimodal understanding of clinical gestures
Department of Control and Computer Engineering, Politecnico di Torino（控制与计算机工程系，都灵理工大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；CNR-IEIIT（意大利国家研究委员会-IEIIT）
MULTISEISMO: A Multimodal Seismic Dataset and Model for Cross-Modal Seismic Understanding
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Oak Ridge National Laboratory（橡树岭国家实验室）；Sandia National Laboratory（桑迪亚国家实验室）；North Carolina State University（北卡罗来纳州立大学）
University of Southern California（南加州大学）；Facebook AI
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework…
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China（中国科学院人工智能安全重点实验室，计算技术研究所，中国科学院，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Tsinghua University, Beijing, China（清华大学，北京，中国）
Meta AI Research（元人工智能研究院）；University of Rochester（罗切斯特大学）；University of Maryland（马里兰大学）
Multimodal video-audio-text understanding and generation can benefit from datasets that are narrow but rich. The narrowness allows bite-sized challenges that the research community can make progress on. The richness ensures we are making progress along the core challenges. To this end, we present a large-scale video-audio-text dataset MUGEN, collected using the open-sourced platform game CoinRun [11]. We made substantial modifications to make the game richer by introducing audio and enabling new interactions. We trained RL agents with different objectives to navigate the game and interact with…
Faculty of Computing, Harbin Institute of Technology（计算机学院，哈尔滨工业大学）；Department of Computer Science and Technology, Harbin Institute of Technology（计算机科学与技术系，哈尔滨工业大学）；Harbin Institute of Technology Suzhou Research Institute（哈尔滨工业大学苏州研究院长）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室，深圳，中国）
UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?
Microsoft Research Asia（微软亚洲研究院）；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；Fudan University（复旦大学）；University of Oxford（牛津大学）
University of Minnesota（明尼苏达大学）；Carnegie Mellon University（卡内基梅隆大学）
In this paper, we study a problem of egocentric scene understanding, i.e., predicting depths and surface normals from an egocentric image. Egocentric scene understanding poses unprecedented challenges: (1) due to large head movements, the images are taken from non-canonical viewpoints (i.e., tilted images) where existing models of geometry prediction do not apply; (2) dynamic foreground objects including hands constitute a large proportion of visual scenes. These challenges limit the performance of the existing models learned from large indoor datasets, such as ScanNet and NYUv2, which compris…
SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
Samsung Research（三星研究所）；Center for Humanoid Research, Korea Institute of Science and Technology（人类研究学院，韩国科学技术院）；Department of plastic surgery, College of medicine, Korea University（医学院整形外科部，韩国大学）；Department of orthopedic surgery, College of medicine, Korea University（医学院骨科部，韩国大学）
MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions
VNU University of Engineering and Technology (VNU-UET)（越南工程技术大学）；Lausanne University Hospital (CHUV)（洛桑大学医院）；University of Lausanne (UNIL)（洛桑大学）；Hanoi Medical University（河内医学院）；National Geriatric Hospital, Hanoi（河内国立老年医院）
