RAMQA：检索增强多模态问答的统一框架
RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering
AI总结：
提出RAMQA统一框架，结合学习排序与生成式排列增强技术，用于多模态检索增强问答，在WebQA和MultiModalQA基准上显著优于强基线。
摘要：
多模态检索增强问答（MRAQA）整合文本和图像，已在信息检索（IR）和自然语言处理（NLP）领域引起广泛关注。传统排序方法依赖基于编码器的小型语言模型，这与现代基于解码器的生成式大语言模型（LLMs）不兼容，而后者已推动了各种NLP任务的进展。为弥合这一差距，我们提出RAMQA，一个将学习排序方法与生成式排列增强排序技术相结合的统一框架。我们首先使用LLaVA作为骨干网络训练一个点式多模态排序器。然后，我们应用指令微调来训练一个LLaMA模型，通过一种创新的自回归多任务学习方法对前k个文档进行重排序。我们的生成式排序模型以各种排列方式从文档候选中生成重排序后的文档ID和具体答案。在两个MRAQA基准测试（WebQA和MultiModalQA）上的实验表明，相较于强基线有显著改进，凸显了我们方法的有效性。代码和数据可在以下网址获取：https://github.com/TonyBY/RAMQA
英文摘要：
Multi-modal retrieval-augmented Question Answering (MRAQA), integrating text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks. To bridge this gap, we propose RAMQA, a unified framework combining learning-to-rank methods with generative permutation-enhanced ranking techniques. We first train a pointwise multi-modal ranker using LLaVA as the backbone. Then, we apply instruction tuning to train a LLaMA model for re-ranking the top-k documents using an innovative autoregressive multi-task learning approach. Our generative ranking model generates re-ranked document IDs and specific answers from document candidates in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, highlighting the effectiveness of our approach. Code and data are available at: https://github.com/TonyBY/RAMQA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
