MM-PhyQA：基于多图像思维链提示的多模态物理问答
MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting
AI总结：
针对大语言模型在多步骤多模态物理推理任务上的短板，研究构建了高中级多模态物理问答数据集MM-PhyQA，提出多图像思维链提示技术，训练的LLaVA-1.5 13b取得71.65%的最高测试准确率。
摘要：
尽管大语言模型（LLM）已能在多种任务中达到人类水平的性能，但在有效处理多步骤物理推理任务方面仍面临挑战。为明确现有模型的不足并推动该领域的进一步研究，我们构建了一个全新的数据集MM-PhyQA，其中包含精心编制的高中阶段多模态物理题目。通过评估当前公开可用的LLM在这类题目中（含多模态元素与不含多模态元素两种情况）的性能，我们旨在揭示其能力边界。对于由多模态输入（本题中为图像和文本）构成的问题的答案生成，我们采用GPT-4进行零样本预测，并使用LLaVA（LLaVA与LLaVA-1.5），后者在我们的数据集上进行了微调。为评估仅含文本输入的LLM的性能，我们测试了Mistral-7B与LLaMA2-7b模型的基础版与微调版的表现。我们还展示了全新的多图像思维链（MI-CoT）提示技术的性能：当用该技术训练LLaVA-1.5 13b时，在我们的数据集上测试取得了最优结果，在多数指标上得分更优，在测试集上的最高准确率达71.65%。
英文摘要：
While Large Language Models (LLMs) can achieve human-level performance in various tasks, they continue to face challenges when it comes to effectively tackling multi-step physics reasoning tasks. To identify the shortcomings of existing models and facilitate further research in this area, we curated a novel dataset, MM-PhyQA, which comprises well-constructed, high schoollevel multimodal physics problems. By evaluating the performance of contemporary LLMs that are publicly available, both with and without the incorporation of multimodal elements in these problems, we aim to shed light on their capabilities. For generating answers for questions consisting of multimodal input (in this case, images and text) we employed Zero-shot prediction using GPT-4 and utilized LLaVA (LLaVA and LLaVA-1.5), the latter of which were fine-tuned on our dataset. For evaluating the performance of LLMs consisting solely of textual input, we tested the performance of the base and fine-tuned versions of the Mistral-7B and LLaMA2-7b models. We also showcased the performance of the novel Multi-Image Chain-of-Thought (MI-CoT) Prompting technique, which when used to train LLaVA-1.5 13b yielded the best results when tested on our dataset, with superior scores in most metrics and the highest accuracy of 71.65% on the test set.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
