arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.12926cs.AI
收藏

MM-PhyRLHF: 面向多模态物理问答的强化学习框架

MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering

IIIT Delhi(德里印度信息技术学院)

Janak Kapuriya, Chhavi Kirtani, Apoorv Singh, Jay Saraf, Naman Lal, Jatin Kumar, Adarsh Raj Shivam, Astha Verma, Avinash Anand, Rajiv Ratn Shah

AI总结:

针对大语言模型在含图像的复杂物理选择题解答上的不足,研究提出基于LLaVA的多模态物理问答聊天机器人,结合MM-PhyQA数据集,通过图像字幕生成和RLHF技术提升解题能力与答案质量,并验证了RLHF的效果。

摘要:

大语言模型(LLMs)近期的发展已展现出其在文本摘要、生成等任务中的显著潜力。然而,在解决需要算术计算和良好概念理解的复杂物理问题时,它们往往会遇到困难。此外,许多物理问题包含图像,这些图像承载着理解问题语境所需的重要细节。我们提出了一种基于大语言多模态模型(LMM)的聊天机器人,用于解答多模态物理选择题(MCQs)。为实现领域适配,我们采用包含印度高中阶段多模态物理题的MM-PhyQA数据集。为提升LMM的性能,我们试验了两种技术:RLHF(基于人类反馈的强化学习)和图像字幕生成。在图像字幕生成环节,我们为每张图像中的示意图添加了详细说明,以减少幻觉现象和图像处理错误。受RLHF中排序方法的启发,我们进一步探索了RLHF方法的整合应用,以增强模型类人化的解题能力。RLHF方法将人类反馈融入LLMs的学习过程,相比普通监督微调模型,它能提升模型的解题技巧、真实性和推理能力,减少答案中的幻觉现象并提高质量。我们使用LLaVA开源模型解答多模态物理选择题,并对比了使用与不使用RLHF时的性能表现。

英文摘要:

Recent advancements in LLMs have shown their significant potential in tasks like text summarization and generation. Yet, they often encounter difficulty while solving complex physics problems that require arithmetic calculation and a good understanding of concepts. Moreover, many physics problems include images that contain important details required to understand the problem's context. We propose an LMM-based chatbot to answer multimodal physics MCQs. For domain adaptation, we utilize the MM-PhyQA dataset comprising Indian high school-level multimodal physics problems. To improve the LMM's performance, we experiment with two techniques, RLHF (Reinforcement Learning from Human Feedback) and Image Captioning. In image captioning, we add a detailed explanation of the diagram in each image, minimizing hallucinations and image processing errors. We further explore the integration of Reinforcement Learning from Human Feedback (RLHF) methodology inspired by the ranking approach in RLHF to enhance the human-like problem-solving abilities of the models. The RLHF approach incorporates human feedback into the learning process of LLMs, improving the model's problem-solving skills, truthfulness, and reasoning capabilities, minimizing the hallucinations in the answers, and improving the quality instead of using vanilla-supervised fine-tuned models. We employ the LLaVA open-source model to answer multimodal physics MCQs and compare the performance with and without using RLHF.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑