通过软负采样增强多模态思维链中的语义
Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling
AI总结：
针对多模态思维链中高文本质量低语义合理性的软负推理依据易引发幻觉、难提升准确率的问题，提出结合软负采样与双向间隔损失的SNSE-CoT方法，经ScienceQA实验验证有效。
摘要：
思维链（Chain of Thought, CoT）已被证明对需要复杂推理的问题十分有效。这类问题中有许多同时涉及文本与多模态信息：面对不同模态的输入，模型会先生成推理依据，再借助其回答问题。受幻觉问题影响，生成的那些文本质量高但语义不符合逻辑的软负推理依据，并不总能帮助提升答案准确率。本研究提出一种采用软负采样的推理依据生成方法（SNSE-CoT），以缓解多模态CoT中的幻觉问题。研究采用五种方法生成软负样本，这些样本与原样本文本高度相似但语义不同。研究将双向间隔损失（Bidirectional margin loss, BML）引入仅包含正、负样本的传统对比学习框架中，把软负样本纳入训练。在ScienceQA数据集上开展的大量实验证明了所提方法的有效性。代码与数据已发布于https://github.com/zgMin/SNSE-CoT。
英文摘要：
Chain of thought (CoT) has proven useful for problems requiring complex reasoning. Many of these problems are both textual and multimodal. Given the inputs in different modalities, a model generates a rationale and then uses it to answer a question. Because of the hallucination issue, the generated soft negative rationales with high textual quality but illogical semantics do not always help improve answer accuracy. This study proposes a rationale generation method using soft negative sampling (SNSE-CoT) to mitigate hallucinations in multimodal CoT. Five methods were applied to generate soft negative samples that shared highly similar text but had different semantics from the original. Bidirectional margin loss (BML) was applied to introduce them into the traditional contrastive learning framework that involves only positive and negative samples. Extensive experiments on the ScienceQA dataset demonstrated the effectiveness of the proposed method. Code and data are released at https://github.com/zgMin/SNSE-CoT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
