arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04144cs.AIcs.LG
收藏

OMG-Agent:迈向鲁棒缺失模态生成的解耦粗到细代理工作流

OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows

University of Electronic Science and Technology of China(电子科技大学)

Ruiting Dai, Zheyu Wang, Haoyu Yang, Yihan Liu, Chengzhi Wang, Zekun Zhang, Zishan Huang, Jiaman Cen, Lisi Mo

AI总结:

OMG-Agent通过解耦粗到细的代理工作流,有效解决多模态数据缺失问题,提升生成的鲁棒性和保真度。

摘要:

数据不完整性严重阻碍了多模态系统的可靠性。现有重建方法面临不同的瓶颈:传统参数/生成模型由于过度依赖内部记忆而容易产生幻觉,而检索增强型框架则在检索刚性方面存在困难。关键的是,这些端到端架构本质上受到语义细节纠缠的限制——逻辑推理与信号合成之间的结构性冲突,这会损害保真度。在本文中,我们提出了OMG-Agent,一种新的框架,它从静态映射转变为动态粗到细的代理工作流。通过模仿“先思考后行动”的认知过程,OMG-Agent将任务分解为三个协同阶段:(1)由大语言模型驱动的语义规划器,通过渐进性上下文推理解决输入模糊性,生成确定性的结构化语义计划;(2)非参数化的证据检索器,将抽象语义扎根于外部知识;(3)检索注入执行器,利用检索到的证据作为灵活的特征提示,以克服刚性并合成高质量的细节。在多个基准上的广泛实验表明,OMG-Agent在多个基准上始终优于最先进的方法,即使在极端缺失的情况下也保持鲁棒性,例如在70%缺失率下CMU-MOSI上取得2.6点的提升。

英文摘要:

Data incompleteness severely impedes the reliability of multimodal systems. Existing reconstruction methods face distinct bottlenecks: conventional parametric/generative models are prone to hallucinations due to over-reliance on internal memory, while retrieval-augmented frameworks struggle with retrieval rigidity. Critically, these end-to-end architectures are fundamentally constrained by Semantic-Detail Entanglement -- a structural conflict between logical reasoning and signal synthesis that compromises fidelity. In this paper, we present \textbf{\underline{O}}mni-\textbf{\underline{M}}odality \textbf{\underline{G}}eneration Agent (\textbf{OMG-Agent}), a novel framework that shifts the paradigm from static mapping to a dynamic coarse-to-fine Agentic Workflow. By mimicking a \textit{deliberate-then-act} cognitive process, OMG-Agent explicitly decouples the task into three synergistic stages: (1) an MLLM-driven Semantic Planner that resolves input ambiguity via Progressive Contextual Reasoning, creating a deterministic structured semantic plan; (2) a non-parametric Evidence Retriever that grounds abstract semantics in external knowledge; and (3) a Retrieval-Injected Executor that utilizes retrieved evidence as flexible feature prompts to overcome rigidity and synthesize high-fidelity details. Extensive experiments on multiple benchmarks demonstrate that OMG-Agent consistently surpasses state-of-the-art methods, maintaining robustness under extreme missingness, e.g., a $2.6$-point gain on CMU-MOSI at $70$\% missing rates.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑