MCERF：通过增强检索推进工程文档的多模态大语言模型评估
MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
AI总结：
提出MCERF框架，结合多模态检索器ColPali与大语言模型推理，通过混合查找、视觉文本融合、高推理和自一致性决策等策略，在DesignQA基准上实现平均准确率相对提升41.1%，无需完整规则书摄入即可处理工程文档中的多模态问答。
摘要：
工程规则书和技术标准包含密集文本、表格和插图等多模态信息，对检索增强生成（RAG）系统构成挑战。基于依赖全文摄入和文本检索的DesignQA框架[1]，本工作建立了多模态ColPali增强检索与推理框架（MCERF），该系统将多模态检索器与大语言模型推理相结合，实现从工程文档中准确高效地回答问题。该系统采用ColPali检索文本和视觉信息，并采用多种检索与推理策略：（i）混合查找模式用于显式规则提及，（ii）视觉到文本融合用于图形和表格引导的查询，（iii）高推理大语言模型模式用于复杂的多模态问题，以及（iv）自一致性决策以稳定响应。模块化框架设计为未来的多模态系统提供了可重用模板，无论底层模型架构如何。此外，本工作建立并比较了两种路由方法：单案例路由方法和多智能体系统，两者均动态分配查询到最优管道。在DesignQA基准上的评估表明，该系统在所有任务上的平均准确率相比基线RAG最佳结果相对提升了41.1%，这是多模态和推理密集型任务上的显著改进，且无需完整规则书摄入。这表明视觉语言检索、模块化推理和自适应路由如何在工程用例中实现可扩展的文档理解。
英文摘要：
Engineering rulebooks and technical standards contain multimodal information like dense text, tables, and illustrations that are challenging for retrieval augmented generation (RAG) systems. Building upon the DesignQA framework [1], which relied on full-text ingestion and text-based retrieval, this work establishes a Multimodal ColPali Enhanced Retrieval and Reasoning Framework (MCERF), a system that couples a multimodal retriever with large language model reasoning for accurate and efficient question answering from engineering documents. The system employs the ColPali, which retrieves both textual and visual information, and multiple retrieval and reasoning strategies: (i) Hybrid Lookup mode for explicit rule mentions, (ii) Vision to Text fusion for figure and table guided queries, (iii) High Reasoning LLM mode for complex multi modal questions, and (iv) SelfConsistency decision to stabilize responses. The modular framework design provides a reusable template for future multimodal systems regardless of underlying model architecture. Furthermore, this work establishes and compares two routing approaches: a single case routing approach and a multi-agent system, both of which dynamically allocate queries to optimal pipelines. Evaluation on the DesignQA benchmark illustrates that this system improves average accuracy across all tasks with a relative gain of +41.1% from baseline RAG best results, which is a significant improvement in multimodal and reasoning-intensive tasks without complete rulebook ingestion. This shows how vision language retrieval, modular reasoning, and adaptive routing enable scalable document comprehension in engineering use cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
