arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11403cs.CVcs.CL
收藏

多模态结构化生成:CVPR 第二届 MMFM 挑战赛技术报告

Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report

Franz Louis Cesista

AI总结:

本文提出多模态结构化生成框架,通过对冻结的多模态基础模型输出 logits 施加硬约束,强制模型生成结构化输出并在回答前推理,在 CVPR 第二届 MMFM 挑战赛中验证了轻量级工程方法优于昂贵微调方案。

摘要:

多模态基础模型(MMFMs)在计算机视觉和自然语言处理任务中已展现出强大的性能。然而,在需要这两种模态高度集成的任务(如文档理解)中,其性能会有所下降。此外,与单模态模型相比,微调和部署这些模型需要显著更多的计算资源和工程投入。在本工作中,我们提出了多模态结构化生成框架,该框架通过对输出 logits 直接施加硬约束,迫使(冻结的)MMFMs 以严格结构化的格式生成输出。这种方法不仅确保模型生成下游 API 易于解析和接收的输出,还使我们能够强制模型在回答之前进行推理,从而在无需昂贵微调的情况下显著提升性能。我们通过在 CVPR 第二届 MMFM 挑战赛中取得的具有竞争力的结果证明了该方法的有效性,突出表明精心设计的轻量级工程可以胜过昂贵且复杂的建模方法。我们所有的脚本、部署步骤和评估结果均可通过 https://github.com/leloykun/MMFM-Challenge 获取。

英文摘要:

Multimodal Foundation Models (MMFMs) have demonstrated strong performance in both computer vision and natural language processing tasks. However, their performance diminishes in tasks that require a high degree of integration between these modalities, such as document understanding. Moreover, finetuning these models and deploying them requires significantly more compute and more engineering effort than unimodal models. In this work, we present Multimodal Structured Generation, a framework that forces (frozen) MMFMs to produce outputs in a strictly structured format by applying hard constraints directly to the output logits. This approach not only ensures that the model generates parseable outputs that downstream APIs can easily ingest but also allows us to force the model to reason before answering, which significantly boosts performance without the need for expensive fine-tuning. We demonstrate the effectiveness of our method through competitive results in the CVPR 2nd MMFM Challenge, highlighting that carefully designed lightweight engineering can outperform expensive and complicated modeling approaches. All of our scripts, deployment steps, and evaluation results can be accessed in https://github.com/leloykun/MMFM-Challenge

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑