HarnessIR：利用多模态基础模型实现通用真实世界图像恢复

中文摘要 AI 辅助

真实世界中的低质量图像遭受复杂的混合退化，包括但不限于噪声、模糊、大气效应等。最近的智能体方法通常将真实世界图像恢复（Real-IR）建模为在特定任务单退化恢复模型上的顺序工具调用问题。然而，这种范式从根本上受到限制，因为复杂的真实世界退化无法通过逐个退化干净地消除，而且用于特定任务模型的工具限制了智能体系统的能力。在这项工作中，我们提出了HarnessIR，一个通过利用多模态基础模型（MFM）作为执行器来实现Real-IR的智能体框架。HarnessIR由五个阶段组成：感知与诊断、按需工具调用、提示词组合、执行以及验证驱动的细化。与先前依赖由特定任务模型组装而成的工具链的智能体Real-IR方法不同，HarnessIR将恢复需求、感知诊断和证据输入到一个MFM中，该MFM在单次传递中执行恢复，随后通过验证阶段来确定结果是否需要进一步处理。在我们的框架下，现成的MFM能够出色地处理恢复任务，在广泛使用的MiO100合成基准上取得了最先进的结果。更重要的是，通过利用MFM强大的泛化能力，HarnessIR在以往智能体图像恢复系统常常难以应对的具有挑战性的真实世界场景中，提供了令人信服的恢复质量。代码可在以下网址获取：此https URL。

英文摘要

Real-world low-quality images suffer from complex mixed degradations, including but not limited to noise, blur, atmospheric effects, etc. Recent agentic methods usually model real-world image restoration (Real-IR) as a sequential tool calling problem over task-specific single-degradation restoration models. This paradigm, however, is fundamentally limited because complex real-world degradations cannot be cleanly undone degradation by degradation, and the tool used for task-specific models caps the capability of the agent system. In this work, we present HarnessIR, an agentic framework for Real-IR by harnessing a multimodal foundation model (MFM) as the executor. HarnessIR consists of five stages: perception and diagnosis, on-demand tool invocation, prompt composition, execution, and verification-driven refinement. Unlike prior agentic Real-IR methods that rely on tool chains assembled from task-specific models, HarnessIR feeds the restoration requirements, the perceptual diagnosis, and the evidence into an MFM that performs restoration in a single pass, followed by verification stages to determine whether the result warrants further processing. Under our harness, off-the-shelf MFMs handle restoration tasks remarkably well, achieving state-of-the-art results on the widely used MiO100 synthetic benchmark. More importantly, by exploiting the strong generalization ability of MFMs, HarnessIR delivers compelling restoration quality on challenging real-world scenes where previous agentic IR systems often struggle. Codes is available at https://github.com/PolyU-VCLab/HarnessIR.

发表机构

• The Hong Kong Polytechnic University（香港理工大学）
• OPPO Research Institute（OPPO研究院）

机构由 AI 辅助整理，请以论文原文为准。