Recent advancements in LLMs have shown their significant potential in tasks like text summarization and generation. Yet, they often encounter difficulty while solving complex physics problems that require arithmetic calculation and a good understanding of concepts. Moreover, many physics problems include images that contain important details required to understand the problem's context. We propose an LMM-based chatbot to answer multimodal physics MCQs. For domain adaptation, we utilize the MM-PhyQA dataset comprising Indian high school-level multimodal physics problems. To improve the LMM's performance, we experiment with two techniques, RLHF (Reinforcement Learning from Human Feedback) and Image Captioning. In image captioning, we add a detailed explanation of the diagram in each image, minimizing hallucinations and image processing errors. We further explore the integration of Reinforcement Learning from Human Feedback (RLHF) methodology inspired by the ranking approach in RLHF to enhance the human-like problem-solving abilities of the models. The RLHF approach incorporates human feedback into the learning process of LLMs, improving the model's problem-solving skills, truthfulness, and reasoning capabilities, minimizing the hallucinations in the answers, and improving the quality instead of using vanilla-supervised fine-tuned models. We employ the LLaVA open-source model to answer multimodal physics MCQs and compare the performance with and without using RLHF.
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning
Sun Yat-sen University(中山大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Dark Matter AI Inc.(暗物智能科技有限公司);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering
Faculty of Computing and Information Technology, King Abdulaziz University(计算机与信息科技学院,国王阿卜杜勒阿齐兹大学);College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University (IMSIU)(计算机与信息科学学院,伊玛目穆罕默德·本·萨乌德伊斯兰大学);School of Computing, Macquarie University(计算学院,麦考瑞大学)
M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence(人工智能交叉学科项目);Department of Intelligence and Information(智能与信息系);Samsung Advanced Institute of Technology(三星先进技术研究院)
Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);NiuTrans Research(NiuTrans研究院);School of Software, Northeastern University(东北大学软件学院)
MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering
Yale University, New Haven, CT, USA(耶鲁大学);McGill University, Montreal, QC, Canada(麦吉尔大学);University of Texas Rio Grande Valley, TX, USA(德克萨斯大学里奥格兰德谷分校)
Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
College of Computer Science, Chongqing University(重庆大学计算机学院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Kuaishou Technology(快手科技)
MultiModalQA: Complex Question Answering over Text, Tables and Images
The Allen Institute for AI(艾伦人工智能研究所);Tel-Aviv University(特拉维夫大学);University of Washington(华盛顿大学)
在回答复杂的问题时,人们可以无缝地结合来自视觉,文本和表格来源的信息。虽然近年来对多个证据推理的模型的兴趣激增,但对跨多个模态推理的问答模型的研究相对较少。在本文中,我们提出了MultiModalQA(MMQA):一个具有挑战性的问答数据集,需要联合推理的文本,表格和图像。我们使用一个新的框架来创建MMQA,该框架用于大规模生成复杂的多模态问题,从维基百科获取表格,并使用每个表格中出现的实体附加图像和文本段落。然后,我们定义了一个形式语言,使我们能够采取的问题,可以回答从一个单一的模态,并将它们结合起来,以产生跨模态的问题。最后,众包工作人员将这些自动生成的问题转化为更流畅的语言。我们通过这个过程创建了29,918个问题,并以经验证明了多模态多跳方法解决我们任务的必要性:我们的多跳模型ImplicitDecomp在跨模态问题上的平均F1为51.7,大大超过了达到38.2 F1的强大基线,但仍然明显落后于人类表现,这是90.1 F1
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
OPPO AI Center, OPPO Inc. China(OPPO AI中心,OPPO公司);The Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Nanyang Technological University, Singapore(新加坡南洋理工大学)
HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering
Human-inspired AI Research, Korea University(韩国大学人机智能研究部);Computer Science and Engineering, Konkuk University(韩国康科大学计算机科学与工程系);Department of Computer Science and Engineering, Korea University(韩国大学计算机科学与工程系)
MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
Key Laboratory of Target Cognition and Application Technology (TCAT), AIRI, CAS(目标认知与应用技术重点实验室(TCAT),空气动力研究所,中国科学院);School of Electronic, Electrical and Communication Engineering, UCAS(电子、电气与通信工程学院,中国科学院大学);Aerospace Information Research Institute, Chinese Academy of Sciences(航天信息研究所,中国科学院);Baidu Inc.(百度公司)