Face video quality assessment (FVQA) deserves to be explored in addition to general video quality assessment (VQA), as face videos are the primary content on social media platforms and human visual system (HVS) is particularly sensitive to human faces. However, FVQA is rarely explored due to the lack of large-scale FVQA datasets. To fill this gap, we present the first large-scale in-the-wild FVQA dataset, FVQ-20K, which contains 20,000 in-the-wild face videos together with corresponding mean opinion score (MOS) annotations. Along with the FVQ-20K dataset, we further propose a specialized FVQA method named FVQ-Rater to achieve human-like rating and scoring for face video, which is the first attempt to explore the potential of large multimodal models (LMMs) for the FVQA task. Concretely, we elaborately extract multi-dimensional features including spatial features, temporal features, and face-specific features (i.e., portrait features and face embeddings) to provide comprehensive visual information, and take advantage of the LoRA-based instruction tuning technique to achieve quality-specific fine-tuning, which shows superior performance on both FVQ-20K and CFVQA datasets. Extensive experiments and comprehensive analysis demonstrate the significant potential of the FVQ-20K dataset and FVQ-Rater method in promoting the development of FVQA.
DHQA-4D: Perceptual Quality Assessment of Dynamic 4D Digital Human
Institute of Image Communication and Network Engineering(图像通信与网络工程研究所);Shanghai Jiao Tong University(上海交通大学);USC-SJTU Institute of Cultural and Creative Industry(USC-SJTU文化产业与创意产业研究院)
VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
Hong Kong University of Science and Technology(香港科技大学);Harbin Institute of Technology(哈尔滨工业大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学)
Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis
Applied Robotics and AI Solutions (ARAS)(应用机器人与人工智能解决方案);Faculties of Electrical and Computer Engineering(电气与计算机工程学院);K.N. Toosi University of Technology(卡里姆·纳尼·托西技术大学);Translational Ophthalmology Research Center(转化眼科研究中心);Farabi Eye Hospital(法拉比眼科医院);Tehran University of Medical Sciences(德黑兰医科大学);Noor Ophthalmology Research Center(努尔眼科研究中心);University of Alberta(阿尔伯塔大学);Departments of Electrical and Computer Engineering & Biomedical Engineering(电气与计算机工程及生物医学工程系)
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University(上海交通大学图像通信与信息处理研究所);MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院(教育部人工智能重点实验室));Huawei Technologies(华为技术有限公司)
Shanghai Jiaotong University(上海交通大学);Media Experience and Evaluation Lab(媒体体验与评估实验室);Nanyang Technological University(南洋理工大学);East China Normal University(华东师范大学)
VersusQ: Pairwise Margin Reasoning for Generalizable Video Quality Assessment
The Chinese University of Hong Kong(香港中文大学);WeChat Vision, Tencent Inc.(腾讯公司视觉部门);Beijing Normal University(北京师范大学);Peking University(北京大学);The University of Hong Kong(香港大学)
Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models
University of Rochester(罗切斯特大学);Northwestern University(西北大学);CMU(卡内基梅隆大学);UCSB(加州大学圣塔芭芭拉分校);Purdue University(普渡大学);UCLA(加州大学洛杉矶分校);University of Oxford(牛津大学);Brown University(布朗大学);University of Virginia(弗吉尼亚大学);Sony Group Corporation(索尼集团公司)
Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare
City University of Hong Kong(香港城市大学);Shanghai Jiao Tong University(上海交通大学);Jiangxi University of Finance and Economics(江西财经大学);Nanyang Technological University(南洋理工大学)
RISE-Video: Can Video Generators Decode Implicit World Rules?
Shanghai Jiao Tong University(上海交通大学);Tencent Youtu Lab(腾讯优图实验室);Xidian University(西安电子科技大学);Beijing Normal University(北京师范大学);The Chinese University of Hong Kong(香港中文大学)
Fudan University(复旦大学);East China Normal University(华东师范大学);Shanghai Jiao Tong University(上海交通大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Texas A&M University(德克萨斯农工大学)
AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment
Institute of Image Communication;Network Engineering, Shanghai Key Laboratory of Digital Media Processing;Transmissions, Shanghai Jiao Tong University, Shanghai;School of Communication \& Electronic Engineering, East China Normal University, Shanghai