Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO) have significantly advanced the reasoning capabilities of large language models. Extending these methods to multimodal settings, however, faces a critical challenge: the instability of std-based normalization, which is easily distorted by extreme samples with nearly positive or negative rewards. Unlike pure-text LLMs, multimodal models are particularly sensitive to such distortions, as both perceptual and reasoning errors influence their responses. To address this, we characterize each sample by its difficulty, defined through perceptual complexity (measured via visual entropy) and reasoning uncertainty (captured by model confidence). Building on this characterization, we propose difficulty-aware group normalization (Durian), which re-groups samples by difficulty levels and shares the std within each group. Our approach preserves GRPO's intra-group distinctions while eliminating sensitivity to extreme cases, yielding significant performance gains across multiple multimodal reasoning benchmarks.
Multi Modal Adaptive Normalization for Audio to Video Generation
Hike(Hike公司)
语音驱动的人脸视频生成是一个复杂的问题,由于其多模态方面,即音频和视频域。音频包括许多基本特征,例如表情、音高、响度、韵律(说话风格),并且面部视频在头部运动、眨眼、嘴唇同步和各种面部动作单元的运动以及时间平滑度方面具有许多可变性。对于生成对抗网络来说,从音频输入和静态图像中合成高度表达的面部视频仍然是一项具有挑战性的任务。在本文中,我们提出了一种多模态自适应归一化(MAN)的基础架构,合成一个说话的人的视频的任意长度作为输入:一个音频信号和一个人的单个图像。该架构使用多模态自适应归一化、关键点热图预测器、光流预测器和基于类激活图[58]的层来学习表情面部组件的运动,从而生成给定人的高度表情化的讲话头部视频。多模态自适应归一化使用音频和视频的各种特征(诸如来自音频信号的Mel频谱图、音高、能量和预测的关键点热图/光流)和单个图像来学习相应的仿射参数以生成高度表达的视频。实验评估表明,与使用GANs的真实语音驱动面部动画(RSDGAN)[53],Speech 2 Vid [10]和其他方法相比,所提出的方法在多个定量指标上具有优异的性能,包括:SSIM(结构相似性指数),PSNR(峰值信噪比),CPBD(图像清晰度),WER(单词出错率),眨眼/秒和LMD(地标距离)。此外,定性评估和在线图灵测试证明了我们的方法的有效性。
Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning
Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education(北京交通大学计算机与信息技术学院;轨道交通控制与安全国家重点实验室;交通大数据与人工智能教育部重点实验室);State Key Laboratory of Advanced Rail Autonomous Operation(格拉斯哥大学心理与神经科学学院);School of Computer and Information Technology, Beijing Jiaotong University;School of Psychology & Neuroscience, University of Glasgow
On the Limits of Multi-modal Meta-Learning with Auxiliary Task Modulation Using Conditional Batch Normalization
University of Edinburgh(爱丁堡大学);Université de Montréal(蒙特利尔大学);Google Research(谷歌研究院);Mila(米拉计算科学研究所);McGill University(麦吉尔大学);University of Calgary(卡尔加里大学);CIFAR(加拿大高级研究所)
RegBN: Batch Normalization of Multimodal Data with Regularization
Technical University of Munich (TUM)(慕尼黑工业大学(TUM));Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Lab for AI in Medical Imaging (AI-Med)(医学影像人工智能实验室)
Multi-Modulation Network for Audio-Visual Event Localization
University of Science and Technology of China(中国科学技术大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Rochester(罗切斯特大学)
Difficulty-Aware Statistical Matching for Dataset Distillation
Zhejiang University(浙江大学);Hokkaido University(北海道大学);The University of Queensland(昆士兰大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);National University of Singapore(新加坡国立大学);Dalian University of Technology(大连理工大学)
AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation
Shanghai AI Laboratory(上海人工智能实验室);The Hong Kong Polytechnic University(香港理工大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);Zhejiang University(浙江大学)
Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Duke University(杜克大学);The University of Hong Kong(香港大学);Peking University(北京大学);University of Chicago(芝加哥大学);Sun Yat-sen University(中山大学)
Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);NLPR&MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Peking University(北京大学);Zhongguancun Academy(中关村学院)
DGFusion: Dual-guided Fusion for Robust Multi-Modal 3D Object Detection
School of Computer Science and Technology, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University(计算机科学与技术学院、交通数据挖掘与具身智能北京市重点实验室、北京交通大学);State Key Laboratory of Internet of Things for Smart City and Department of Electrome chanical Engineering, University of Macau(智能城市物联网国家重点实验室、澳门大学机电工程系);Fujian Key Laboratory of Sensing and Computing for Smart Cities, Xiamen University(智能城市感知与计算福建省重点实验室、厦门大学);School of Mechanical and Aerospace Engineering, Nanyang Technological University(机械与航空航天工程学院、南洋理工大学);State Key Laboratory of Robotics and System, Harbin Institute of Technology(机器人系统国家重点实验室、哈尔滨工业大学)
Partially Supervised Unpaired Multi-Modal Learning for Label-Efficient Medical Image Segmentation
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(高性能计算研究所(A*STAR));The Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University(山东大学-新加坡南洋理工大学人工智能联合研究中心)
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
AI Singapore, National University of Singapore(新加坡国立大学新加坡人工智能);School of Computing, National University of Singapore(新加坡国立大学计算学院);Institute of Advanced Intelligence and Computing, A*STAR(新加坡科技研究局先进智能与计算研究所)
ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
Alibaba Group(阿里巴巴集团);Beijing Institute of Technology(北京理工大学);Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Zhongguancun Academy(中关村学院)
Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与工程学院);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算学系);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Multi-modal Auto-regressive Modeling via Visual Words
School of Computer Science, Wuhan University(武汉大学计算机学院);Shanghai Jiao Tong University(上海交通大学);School of Information Management, Wuhan University(武汉大学信息管理学院)