用于多模态表示学习的思维图软提示
Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learning
AI总结：
针对线性思维链难以刻画人类非线性、多方面思维的问题，提出 AGoT 聚合思维图软提示机制，并验证其在多类多模态任务和领域泛化上的有效性。
摘要：
思维链技术已在多模态任务中获得良好反响。它是一种逐步的线性推理过程，通过调整链的长度来提升所生成提示的性能。然而，人类的思维过程主要是非线性的，因为这些过程同时涵盖多个方面，并采用动态调整与更新机制。因此，我们提出一种新颖的 Aggregation-Graph-of-Thought（AGoT，聚合思维图）机制，用于多模态表示学习中的软提示调优。所提出的 AGoT 不仅将人类思维过程建模为一条链，还将每一步建模为一个推理聚合图，以应对单步推理中被忽视的多个思维方面。这使整个推理过程转化为提示聚合与提示流操作。实验表明，我们借助 AGoT 软提示增强的多模态模型在文本-图像检索、视觉问答和图像识别等多项任务中取得了良好结果。此外，我们证明由于其更优的推理能力，该模型具有良好的领域泛化性能。
英文摘要：
The chain-of-thought technique has been received well in multi-modal tasks. It is a step-by-step linear reasoning process that adjusts the length of the chain to improve the performance of generated prompts. However, human thought processes are predominantly non-linear, as they encompass multiple aspects simultaneously and employ dynamic adjustment and updating mechanisms. Therefore, we propose a novel Aggregation-Graph-of-Thought (AGoT) mechanism for soft-prompt tuning in multi-modal representation learning. The proposed AGoT models the human thought process not only as a chain but also models each step as a reasoning aggregation graph to cope with the overlooked multiple aspects of thinking in single-step reasoning. This turns the entire reasoning process into prompt aggregation and prompt flow operations. Experiments show that our multi-modal model enhanced with AGoT soft-prompting achieves good results in several tasks such as text-image retrieval, visual question answering, and image recognition. In addition, we demonstrate that it has good domain generalization performance due to better reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
