LMM-VQA：用大型多模态模型推进视频质量评估
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
AI总结：
提出首个大型多模态视频质量评估模型LMM-VQA，通过时空视觉建模与问答指令微调提取并对齐质量特征，在五个VQA基准上达到SOTA且泛化能力平均提升5%。
摘要：
流媒体平台上视频的爆炸式增长凸显了对有效视频质量评估（VQA）算法的迫切需求，以监控并感知优化流媒体视频质量。然而，由于视频内容多样以及复杂的空间和时间失真，VQA仍是一项极具挑战性的任务，因此需要更先进的方法来解决这些问题。如今，如GPT-4V等大型多模态模型（LMMs）在各种视觉理解任务中展现出强大能力，促使我们利用LMMs强大的多模态表示能力来解决VQA任务。因此，我们提出了首个大型多模态视频质量评估（LMM-VQA）模型，该模型引入了一种新颖的时空视觉建模策略，用于质量感知特征提取。具体而言，我们首先将质量回归问题重新表述为问答（Q&A）任务，并为VQA指令微调构建Q&A提示。然后，我们设计了一个时空视觉编码器来提取空间和时间特征以表示视频的质量特征，随后这些特征通过时空投影器映射到语言空间以进行模态对齐。最后，将对齐的视觉标记和质量查询文本标记聚合，作为大型语言模型（LLM）的输入，以生成质量分数和等级。大量实验表明，LMM-VQA在五个VQA基准上实现了最先进的性能，在泛化能力上较现有方法平均提升5%。此外，得益于时空编码器和投影器的先进设计，LMM-VQA在通用视频理解任务上也表现异常出色，进一步验证了其有效性。我们的代码将发布于https://github.com/Sueqk/LMM-VQA。
英文摘要：
The explosive growth of videos on streaming media platforms has underscored the urgent need for effective video quality assessment (VQA) algorithms to monitor and perceptually optimize the quality of streaming videos. However, VQA remains an extremely challenging task due to the diverse video content and the complex spatial and temporal distortions, thus necessitating more advanced methods to address these issues. Nowadays, large multimodal models (LMMs), such as GPT-4V, have exhibited strong capabilities for various visual understanding tasks, motivating us to leverage the powerful multimodal representation ability of LMMs to solve the VQA task. Therefore, we propose the first Large Multi-Modal Video Quality Assessment (LMM-VQA) model, which introduces a novel spatiotemporal visual modeling strategy for quality-aware feature extraction. Specifically, we first reformulate the quality regression problem into a question and answering (Q&A) task and construct Q&A prompts for VQA instruction tuning. Then, we design a spatiotemporal vision encoder to extract spatial and temporal features to represent the quality characteristics of videos, which are subsequently mapped into the language space by the spatiotemporal projector for modality alignment. Finally, the aligned visual tokens and the quality-inquired text tokens are aggregated as inputs for the large language model (LLM) to generate the quality score and level. Extensive experiments demonstrate that LMM-VQA achieves state-of-the-art performance across five VQA benchmarks, exhibiting an average improvement of $5\%$ in generalization ability over existing methods. Furthermore, due to the advanced design of the spatiotemporal encoder and projector, LMM-VQA also performs exceptionally well on general video understanding tasks, further validating its effectiveness. Our code will be released at https://github.com/Sueqk/LMM-VQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
