arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03062cs.AI
收藏

图像优先还是文本优先?优化大语言模型提示与推理任务中的模态排序

Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Tasks

School of Mathematical and Computational Sciences(数学与计算科学学院);Massey University(梅西大学)

Grant Wardle, Teo Susnjak

AI总结:

本文通过三个商用大语言模型实证研究图像与文本在多模态提示中的排序对推理性能的影响,发现简单任务中排序影响显著而复杂任务中减弱,并提出模态序列应与推理逻辑流程对齐以改进提示设计。

摘要:

本文研究了多模态提示中图像与文本的排序如何影响大语言模型(LLMs)的推理性能。我们使用三个商用大语言模型进行了实证评估。结果表明,模态呈现的顺序会显著影响性能,尤其是在不同复杂度的任务中。对于涉及单张图像的较简单任务,模态排序对准确率有明显影响。然而,在涉及多张图像和复杂推理步骤的较复杂任务中,排序的影响减弱,这可能是由于任务认知需求的增加所致。我们的发现还强调了问题/提示结构的重要性。在嵌套和多步推理任务中,模态排序在塑造模型性能方面发挥了关键作用。虽然大语言模型在推理的初始阶段表现出色,但它们在重新整合早期信息方面存在困难,这凸显了Transformer架构中多跳推理的挑战。这表明,使模态序列与推理步骤的逻辑流程保持一致,比单纯的模态顺序更为关键。这些见解为改进多模态提示设计提供了有价值的启示,并在教育、医学影像和跨模态学习等领域具有更广泛的应用前景。

英文摘要:

This paper examines how the sequencing of images and text within multi-modal prompts influences the reasoning performance of large language models (LLMs). We performed empirical evaluations using three commercial LLMs. Our results demonstrate that the order in which modalities are presented can significantly affect performance, particularly in tasks of varying complexity. For simpler tasks involving a single image, modality sequencing had a clear impact on accuracy. However, in more complex tasks involving multiple images and intricate reasoning steps, the effect of sequencing diminished, likely due to the increased cognitive demands of the task. Our findings also highlight the importance of question/prompt structure. In nested and multi-step reasoning tasks, modality sequencing played a key role in shaping model performance. While LLMs excelled in the initial stages of reasoning, they struggled to re-incorporate earlier information, underscoring the challenges of multi-hop reasoning within transformer architectures. This suggests that aligning the sequence of modalities with the logical flow of reasoning steps is more critical than modality order alone. These insights offer valuable implications for improving multi-modal prompt design, with broader applications across fields such as education, medical imaging, and cross-modal learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑