揭示多模态大语言模型中的视觉质量悖论
Demystifying the Visual Quality Paradox in Multimodal Large Language Models
AI总结：
本研究揭示多模态大语言模型存在视觉质量悖论，即低感知质量图像可能提升性能，并提出VQ-TTT轻量级自适应模块，通过低秩核与LoRA微调动态调整输入，显著提升模型准确率。
摘要：
近年来，多模态大语言模型（MLLMs）在基准视觉语言任务上表现出色，但输入视觉质量如何影响其响应仍鲜为人知。更高的图像感知质量是否已转化为更好的MLLM理解？我们开展了首个系统性研究，涵盖领先的MLLMs和一系列视觉语言基准，对每张图像施加受控退化与风格变化。令人惊讶的是，我们发现了视觉质量悖论：当图像偏离人类感知保真度时，模型、任务甚至单个实例的性能可能提升。现成的恢复流程无法调和这些特异性的偏好。为弥合这一差距，我们提出了视觉质量测试时调优（VQ-TTT）——一种轻量级自适应模块，其：（1）在冻结的视觉编码器前插入可学习的低秩核以调节频率内容；（2）通过LoRA仅微调浅层视觉编码器层。VQ-TTT在单次前向传播中动态调整每个输入图像，使其与任务特定的模型偏好对齐。在评估的MLLMs和所有数据集上，VQ-TTT显著提升了平均准确率，且无需外部模型、缓存特征或额外训练数据。这些发现重新定义了MLLMs的“更好”视觉输入，并强调了在AI作为主要数据消费者的新时代，需要自适应而非普遍“干净”的图像。
英文摘要：
Recent Multimodal Large Language Models (MLLMs) excel on benchmark vision-language tasks, yet little is known about how input visual quality shapes their responses. Does higher perceptual quality of images already translate to better MLLM understanding? We conduct the first systematic study spanning leading MLLMs and a suite of vision-language benchmarks, applying controlled degradations and stylistic shifts to each image. Surprisingly, we uncover a visual-quality paradox: model, task, and even individual-instance performance can improve when images deviate from human-perceived fidelity. Off-the-shelf restoration pipelines fail to reconcile these idiosyncratic preferences. To close the gap, we introduce Visual-Quality Test-Time Tuning (VQ-TTT)-a lightweight adaptation module that: (1) inserts a learnable, low-rank kernel before the frozen vision encoder to modulate frequency content; and (2) fine-tunes only shallow vision-encoder layers via LoRA. VQ-TTT dynamically adjusts each input image in a single forward pass, aligning it with task-specific model preferences. Across the evaluated MLLMs and all datasets, VQ-TTT lifts significant average accuracy, with no external models, cached features, or extra training data. These findings redefine ``better'' visual inputs for MLLMs and highlight the need for adaptive, rather than universally ``clean'', imagery, in the new era of AI being the main data customer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
