arXivDaily arXiv每日学术速递 周一至周五更新

大厂专区

Xiaomi(小米)

至 收录 207
2607.17900 2026-07-21 cs.SD 新提交

Harness TTS: Towards Context-Aware Expressive Speech Synthesis with Harness Layer

利用TTS:借助利用层实现上下文感知的富有表现力的语音合成

Shengfan Shen, Di Wu, Xingchen Song, Dinghao Zhou, Pengyu Cheng, Sixiang Lyu, Jian Luan, Shuai Wang

机构 * Xiaomi Inc.(小米公司) Nanjing University(南京大学)

AI总结 研究针对语音助手富有表现力的语音合成中灵活风格控制的需求,提出Harness TTS轻量级控制层,通过封闭集提示 - 工具路由实现风格控制,实验证明其在路由和合成任务中表现出色,为语音助手表达控制提供实用方案。

详情
AI中文摘要

语音助手的富有表现力的语音合成需要灵活的风格控制,以适应明确请求和更广泛的交互上下文。我们提出了Harness TTS,这是一个轻量级控制层,围绕TTS引擎以外部化并管理其表达行为。它将风格控制重新制定为封闭集提示 - 工具路由:离线时,使用结构化元数据构建风格提示工具的紧凑注册表;在线时,一个大语言模型规划器根据优先级感知观察模式选择合适的工具,TTS执行器使用相应的提示音频合成语音。我们在路由和合成任务上评估了Harness TTS。在路由方面,Qwen3 - 4B在显式、隐式和冲突子集上的Top - 1准确率分别为74.3%、43.0%和64.6%。对于合成,在CosyVoice3和VoxCPM2上的实验表明,Harness TTS优于仅指令控制,实现了更高的指令跟随胜率(在CosyVoice3上优势为23.1 - 35.6分,在VoxCPM2上为13.8 - 20.0分),并将UTMOSv2分数提高了0.11 - 0.38。此外,4B规划器在标准模式下不到50毫秒就能给出第一个工具推荐,对实时交互引入的延迟可忽略不计。这些结果表明,为TTS引擎配备专用的利用层为语音助手表达控制提供了一个实用、可审计且上下文感知的解决方案。

英文摘要

Expressive speech synthesis for voice assistants requires flexible style control that adapts to explicit requests and broader interaction context. We propose Harness TTS, a lightweight control layer that wraps around a TTS engine to externalize and govern its expressive behavior. It reformulates style control as closed-set prompt-tool routing: offline, a compact registry of stylistic prompt tools is constructed with structured metadata; online, an LLM planner selects the appropriate tool based on a priority-aware observation schema, and the TTS executor synthesizes speech using the corresponding prompt audio. We evaluate Harness TTS on both routing and synthesis tasks. In routing, Qwen3-4B achieves Top-1 accuracies of 74.3%, 43.0%, and 64.6% on explicit, implicit, and conflict subsets. For synthesis, experiments on CosyVoice3 and VoxCPM2 show that Harness TTS outperforms instruction-only control, achieving higher instruction-following win rates (margins of 23.1-35.6 points on CosyVoice3 and 13.8-20.0 points on VoxCPM2) and improving UTMOSv2 scores by 0.11-0.38. Moreover, the 4B planner delivers its first tool recommendation in under 50 ms in standard mode, introducing negligible latency for real-time interaction. These results demonstrate that equipping TTS engines with a dedicated Harness layer offers a practical, auditable, and context-aware solution for voice assistant expression control.

URL PDF HTML 收藏
2607.17521 2026-07-21 cs.RO 新提交

GeoWorldAD: Geometry World Action Model for Autonomous Driving

GeoWorldAD:用于自动驾驶的几何世界行动模型

Songyan Zhang, Jinyuan Tian, Hanbing Li, Daqi Liu, Hao Chen, Wenhui Huang, Fang Li, Guang Chen, Hangjun Ye, Long Chen, Kuiyuan Yang, Chen Lv

机构 * Nanyang Technological University(南洋理工大学) Xiaomi EV(小米汽车) Zhejiang University(浙江大学) Harvard University(哈佛大学)

AI总结 研究自动驾驶中安全高效规划决策问题,提出GeoWorldAD模型,通过在自我对齐3D空间中规划轨迹、用潜在未来几何标记预测场景演变,并逐步聚合多尺度几何线索,实验证明该模型在自动驾驶方面性能先进。

详情
AI中文摘要

自动驾驶需要在动态3D环境中做出安全且高效的规划决策。尽管近期的视觉/视频行动模型能直接从视觉观察中学习策略并随视觉Transformer和大规模训练数据发展良好,但常缺乏明确的几何基础和对未来的空间引导。本文提出GeoWorldAD,一种在自我对齐的3D空间中进行轨迹规划并通过潜在未来几何标记预测短视距场景演变的几何世界行动模型。当前几何为安全规划提供基本空间约束,未来几何揭示周围物体和以自我为中心的自由空间如何演变,减少过度保守决策且不牺牲安全性。通过迭代轨迹细化逐步聚合多尺度当前几何和潜在未来几何以有效利用这些几何线索。在NAVSIM v1和v2上的实验证明了其先进性能,凸显了明确的3D几何基础和未来几何世界建模对安全高效自动驾驶的有效性。

英文摘要

Autonomous driving requires both safe and efficient planning decisions in dynamic 3D environments. Although recent Vision/Video-Action models learn policies directly from visual observations and scale well with advances in vision transformers and large-scale training data, they often lack explicit geometric grounding and future-aware spatial guidance, limiting their ability to balance collision avoidance and driving progress. In this work, we propose GeoWorldAD, a geometry world action model that grounds trajectory planning in ego-aligned 3D space and anticipates short-horizon scene evolution with latent future geometry tokens. Present geometry provides essential spatial constraints for safe planning, while future geometry reveals how surrounding agents and ego-centric free space may evolve, reducing overly conservative decisions without sacrificing safety. To efficiently exploit these geometric cues, GeoWorldAD progressively aggregates multi-scale present geometry and latent future geometry through iterative trajectory refinement. Experiments on NAVSIM v1 and v2 demonstrate state-of-the-art performance, highlighting the effectiveness of explicit 3D geometry grounding and future geometry world modeling for safe and efficient autonomous driving.

URL PDF HTML 收藏
2602.11961 2026-07-21 cs.CL 版本更新

Scaling Model and Data for Multilingual Machine Translation with Open Large Language Models

基于开放大语言模型的多语言机器翻译模型与数据扩展

Yuzhe Shang, Pengzhi Gao, Wei Liu, Jian Luan, Jinsong Su

机构 * School of Informatics, Xiamen University, China(厦门大学信息学院) MiLM Plus, Xiaomi Inc., Beijing, China(小米公司)

AI总结 本文提出MiLMMT-46模型,通过模型和数据扩展提升多语言机器翻译性能,超越现有SOTA模型并接近Google Translate等强 proprietary 系统。

详情
AI中文摘要

开放大语言模型(LLMs)近年来在多语言能力方面展现出提升的潜力。本文研究了开放LLMs在多语言机器翻译(MT)中的应用,涵盖了多种语言,并探讨了在通过持续预训练和指令微调将开放LLMs适应于多语言MT时,模型扩展和数据扩展的影响。基于Gemma3模型家族,我们开发了MiLMMT-46,该模型在46种语言上实现了顶级的多语言翻译性能。大量实验表明,MiLMMT-46在性能上 consistently 超过了最近的最先进(SOTA)模型,包括Seed-X、HY-MT-1.5和TranslateGemma,并在与强 proprietary 系统如Google Translate和Gemini 3 Pro的性能上具有竞争力。模型已发布至https://huggingface.co/collections/xiaomi-research/milmmt-46。代码已发布至https://github.com/xiaomi-research/gemmax。

英文摘要

Open large language models (LLMs) have demonstrated improving multilingual capabilities in recent years. In this paper, we present a study of open LLMs for multilingual machine translation (MT) across a range of languages, and investigate the effects of model scaling and data scaling when adapting open LLMs to multilingual MT through continual pretraining and instruction finetuning. Based on the Gemma3 model family, we develop MiLMMT-46, which achieves top-tier multilingual translation performance across 46 languages. Extensive experiments show that MiLMMT-46 consistently outperforms recent state-of-the-art (SOTA) models, including Seed-X, HY-MT-1.5, and TranslateGemma, and achieves competitive performance with strong proprietary systems such as Google Translate and Gemini 3 Pro. Models are released at https://huggingface.co/collections/xiaomi-research/milmmt-46. Codes are released at https://github.com/xiaomi-research/gemmax.

URL PDF HTML 收藏
2511.14592 2026-07-21 cs.RO cs.AI 版本更新

DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Risks

DSBench:用于评估外部和车内风险的综合基准测试

Xianhui Meng, Yuchen Zhang, Zhijian Huang, Zheng Lu, Ziling Ji, Yandan Lin, Yaoyao Yin, Hongyuan Zhang, Wei Zhou, Guangfeng Jiang, Li Zhang, Long Chen, Hangjun Ye, Jun Liu, Xiaoshuai Hao

机构 * University of Science and Technology of China(中国科学技术大学) Georgia Institute of Technology(佐治亚理工学院) Xiaomi EV(小米电动车) Fudan University(复旦大学) Xidian University(西安电子科技大学) South China University of Technology(华南理工大学) The University of Hong Kong(香港大学)

AI总结 针对视觉语言模型在安全关键场景适用性未充分探索的问题,引入DSBench综合基准测试,涵盖外部与车内风险,分多类别。评估发现模型在复杂情况下降,构建数据集微调可提升性能,推动自动驾驶技术发展,相关资源将公开。

详情
AI中文摘要

视觉语言模型(VLMs)在自动驾驶方面前景广阔,但在安全关键场景中的适用性尚未充分探索,引发安全担忧。由于缺乏同时评估外部环境风险和车内驾驶行为安全的综合基准测试,我们引入了DSBench,首个用于统一评估VLM对各种安全风险认知的综合驾驶安全基准测试。它涵盖外部环境风险和车内驾驶行为安全两大类别,分为10个关键类别和28个子类别。对多种主流开源和闭源VLM的广泛评估显示,在复杂安全关键情况下性能显著下降。为此,我们构建了一个包含98K实例的大型数据集,表明在此数据集上微调可显著提高现有VLM的安全性能,为推进自动驾驶技术铺平道路。基准测试工具包、代码和模型检查点将公开可用。

英文摘要

Vision-Language Models (VLMs) show great promise for autonomous driving, but their suitability for safety-critical scenarios is largely unexplored, raising safety concerns. This issue arises from the lack of comprehensive benchmarks that assess both external environmental risks and in-cabin driving behavior safety simultaneously. To bridge this critical gap, we introduce DSBench, the first comprehensive Driving Safety Benchmark designed to assess a VLM's awareness of various safety risks in a unified manner. DSBench encompasses two major categories: external environmental risks and in-cabin driving behavior safety, divided into 10 key categories and a total of 28 sub-categories. This comprehensive evaluation covers a wide range of scenarios, ensuring a thorough assessment of VLMs' performance in safety-critical contexts. Extensive evaluations across various mainstream open-source and closed-source VLMs reveal significant performance degradation under complex safety-critical situations, highlighting urgent safety concerns. To address this, we constructed a large dataset of 98K instances focused on in-cabin and external safety scenarios, showing that fine-tuning on this dataset significantly enhances the safety performance of existing VLMs and paves the way for advancing autonomous driving technology. The benchmark toolkit, code, and model checkpoints will be publicly accessible.

URL PDF HTML 收藏
2607.15330 2026-07-20 cs.RO cs.CV 新提交

Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories

小米机器人1:利用超过10万小时的真实世界轨迹扩展视觉语言动作模型

Xiaomi Robotics Team, Jun Guo, Piaopiao Jin, Jason Li, Peiyan Li, Yingyan Li, Futeng Liu, Wanli Peng, Optimus Qin, Yifei Su, Nan Sun, Qiao Sun, Runze Suo, Heyun Wang, Yunhong Wang, Rujie Wu, Caoyu Xia, Lina Zhang, Jack Zhao, Guoliang Chen, Wenlong Chen, Xinze He, Bin Li, Qing Li, Zhuorong Li, Heng Qu, Wenxuan Song, Diyun Xiang, Yifan Xie, Peiran Xu, Hangjun Ye, Wen Ye, Han Zhao, Quanyun Zhou

机构 * Xiaomi Robotics(小米机器人)

AI总结 研究提出小米机器人1这一视觉语言动作模型,采用两阶段训练方法,预训练利用大量真实轨迹赋予模型能力,后训练使其与机器人实体及指令对齐。该模型在多基准测试中表现优异,能有效微调,建立新的最先进水平。

Comments Project page: https://robotics.xiaomi.com/xiaomi-robotics-1.html

详情
AI中文摘要

我们展示了小米机器人1,这是一个基础的视觉语言动作(VLA)模型,能够:(1)遵循各种语言指令,在未见环境中开箱即用执行广泛的移动操作任务;(2)使用最少的微调数据有效适应新的下游任务。我们提出了一个由预训练和后训练组成的两阶段训练方法。预训练期间,通过在超过10万小时通过UMI设备收集的真实世界操作轨迹上训练,赋予模型广泛且可泛化的动作生成能力。关键的是,我们开发了一个可扩展的自动标注管道,用描述场景状态转换的自然语言标注轨迹片段,为动作学习提供丰富而精确的条件。后训练旨在使这些能力与机器人实体以及人类自然用于提示机器人的命令式指令对齐。大量实验证明了强大的扩展行为。小米机器人1在预训练期间随着数据规模和模型大小的增加持续改进。这种扩展行为直接转移到后训练中,更强的预训练模型在未见环境中产生更好的开箱即用真实机器人性能。此外,小米机器人1作为一个强大的机器人基础策略,可以在复杂、灵巧的任务上以高数据效率进行有效微调。在多个模拟基准测试中,小米机器人1优于现有方法。值得注意的是,它在RoboCasa365上以57.6%的成功率建立了新的最先进水平,超过了之前的最佳成绩46.6%。此外,它在RoboDojo上的平均得分为20.07,显著优于先前的最先进水平(13.07)。代码和模型检查点将被发布。项目页面:这个https链接

英文摘要

We present Xiaomi-Robotics-1, a foundational vision-language-action (VLA) model capable of (1) following diverse language instructions to perform a wide range of mobile manipulation tasks in unseen environments out-of-the-box, and (2) efficiently adapting to novel downstream tasks with minimal fine-tuning data. We propose a two-stage training recipe consisting of pre-training and post-training. During pre-training, we imbue the model with broad and generalizable action-generation capabilities by training on over 100k hours of real-world manipulation trajectories collected via UMI devices. Crucially, we develop a scalable auto-labeling pipeline that annotates trajectory clips with natural languages describing scene state transitions, providing rich and precise conditioning for action learning. During post-training, we aim to align these capabilities with robot embodiments and imperative instructions that humans naturally use to prompt robots. Extensive experiments demonstrate strong scaling behavior. Xiaomi-Robotics-1 consistently improves with increased data scales and model sizes during pre-training. This scaling behavior directly transfers to post-training, where a stronger pre-training model yields better out-of-the-box real-robot performance in unseen environments. Furthermore, Xiaomi-Robotics-1 serves as a strong robot foundation policy that can be efficiently fine-tuned on complex, dexterous tasks with high data efficiency. Across multiple simulation benchmarks, Xiaomi-Robotics-1 outperforms state-of-the-art methods. Notably, it establishes a new state-of-the-art with a 57.6% success rate on RoboCasa365, surpassing the previous best of 46.6%. Furthermore, it achieves an average score of 20.07 on RoboDojo, significantly outperforming the prior state-of-the-art (13.07). Code and model checkpoints will be released. Project page: https://robotics.xiaomi.com/xiaomi-robotics-1.html

URL PDF HTML 收藏
2603.12262 2026-07-20 cs.CV 版本更新

Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously

视频流思考:VideoLLMs可以同时观看和思考

Yiran Guan, Liang Yin, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, Xiang Bai

机构 * Huazhong University of Science and Technology(华中科技大学) MiLM Plus Xiaomi Inc.(小米公司)

AI总结 VST提出了一种视频流理解的新范式,通过在流式过程中激活推理,提高实时响应和理解能力,同时在多个基准测试中表现出更高的效率和泛化能力。

Comments Accepted by ECCV 2026, project page https://1ranguan.github.io/VST/

详情
AI中文摘要

在线视频大语言模型(VideoLLMs)在支持响应式、实时交互中发挥关键作用。现有方法专注于流式感知,缺乏同步的逻辑推理流。然而,直接应用测试时缩放方法会导致不可接受的响应延迟。为了解决这一权衡,我们提出了视频流思考(VST),一种新的流式视频理解范式。它支持一边观看一边思考的机制,在流式过程中激活对输入视频片段的推理。这种设计通过在视频播放过程中摊销LLM推理延迟,提高了及时理解和连贯认知,同时保持实时响应能力。此外,我们引入了一个全面的训练后流水线,该流水线整合了VST-SFT,通过结构上适应离线VideoLLM到因果流式推理,以及VST-RL,通过在多轮视频交互环境中进行自我探索提供端到端改进。此外,我们设计了一个自动化训练数据合成流水线,利用视频知识图谱生成高质量的流式问答对,使用实体-关系基础的流式推理链来强制多证据推理和持续关注视频流。广泛评估显示,VST-7B在在线基准测试中表现强劲,例如在StreamingBench上达到79.5%的准确率,在OVO-Bench上达到59.3%的准确率。同时,VST在离线长格式或推理基准测试中仍具竞争力。与Video-R1相比,VST响应速度提高了15.7倍,并在VideoHolmes上实现了+5.4%的改进,证明了其在多样化视频理解任务中的更高效率和强大泛化能力。代码、数据和模型将在https://github.com/1ranGuan/VST上发布。

英文摘要

Online Video Large Language Models (VideoLLMs) play a critical role in supporting responsive, real-time interaction. Existing methods focus on streaming perception, lacking a synchronized logical reasoning stream. However, directly applying test-time scaling methods incurs unacceptable response latency. To address this trade-off, we propose Video Streaming Thinking (VST), a novel paradigm for streaming video understanding. It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming. This design improves timely comprehension and coherent cognition while preserving real-time responsiveness by amortizing LLM reasoning latency over video playback. Furthermore, we introduce a comprehensive post-training pipeline that integrates VST-SFT, which structurally adapts the offline VideoLLM to causal streaming reasoning, and VST-RL, which provides end-to-end improvement through self-exploration in a multi-turn video interaction environment. Additionally, we devise an automated training-data synthesis pipeline that uses video knowledge graphs to generate high-quality streaming QA pairs, with an entity-relation grounded streaming Chain-of-Thought to enforce multi-evidence reasoning and sustained attention to the video stream. Extensive evaluations show that VST-7B performs strongly on online benchmarks, e.g. 79.5% on StreamingBench and 59.3% on OVO-Bench. Meanwhile, VST remains competitive on offline long-form or reasoning benchmarks. Compared with Video-R1, VST responds 15.7 times faster and achieves +5.4% improvement on VideoHolmes, demonstrating higher efficiency and strong generalization across diverse video understanding tasks. Code, data, and models will be released at https://github.com/1ranGuan/VST.

URL PDF HTML 收藏
2512.13636 2026-07-20 cs.CV cs.RO 版本更新

MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learning

MindDrive: 一种通过在线强化学习实现自动驾驶的视觉-语言-动作模型

Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye, Dingkang Liang, Xiang Bai

机构 * Huazhong University of Science and Technology(华中科技大学) Xiaomi EV(小米电动车)

AI总结 MindDrive通过在线强化学习实现自动驾驶中的视觉-语言-动作模型,结合决策专家和动作专家,提升复杂场景下的驾驶决策与探索效率。

Comments Accepted by ECCV 2026; Project Page: https://xiaomi-mlab.github.io/MindDrive/

详情
AI中文摘要

当前自动驾驶中的视觉-语言-动作(VLA)范式主要依赖于模仿学习(IL),这引入了固有的挑战,如分布偏移和因果混淆。在线强化学习通过试错学习提供了一条解决这些问题的有希望的途径。然而,将在线强化学习应用于自动驾驶中的VLA模型受到连续动作空间中探索效率低的阻碍。为克服这一限制,我们提出了MindDrive,一种由大型语言模型(LLM)和两组不同的LoRA参数组成的VLA框架。一个LLM用作决策专家进行场景推理和驾驶决策,而另一个则作为动作专家,动态地将语言决策映射到可行的轨迹中。通过将轨迹级奖励反馈到推理空间中,MindDrive能够在有限的离散语言驾驶决策集上实现试错学习,而不是直接在连续动作空间中操作。这种方法有效平衡了复杂场景中的最优决策、人类驾驶行为和在线强化学习中的高效探索。使用轻量级的Qwen-0.5B LLM,MindDrive在具有挑战性的Bench2Drive基准上实现了78.04的驾驶得分(DS)和55.09%的成功率(SR)。据我们所知,这是首次展示在线强化学习在自动驾驶VLA模型中的有效性的工作。

英文摘要

Current Vision-Language-Action (VLA) paradigms in autonomous driving primarily rely on Imitation Learning (IL), which introduces inherent challenges such as distribution shift and causal confusion. Online Reinforcement Learning offers a promising pathway to address these issues through trial-and-error learning. However, applying online reinforcement learning to VLA models in autonomous driving is hindered by inefficient exploration in continuous action spaces. To overcome this limitation, we propose MindDrive, a VLA framework comprising a large language model (LLM) with two distinct sets of LoRA parameters. The one LLM serves as a Decision Expert for scenario reasoning and driving decision-making, while the other acts as an Action Expert that dynamically maps linguistic decisions into feasible trajectories. By feeding trajectory-level rewards back into the reasoning space, MindDrive enables trial-and-error learning over a finite set of discrete linguistic driving decisions, instead of operating directly in a continuous action space. This approach effectively balances optimal decision-making in complex scenarios, human-like driving behavior, and efficient exploration in online reinforcement learning. Using the lightweight Qwen-0.5B LLM, MindDrive achieves Driving Score (DS) of 78.04 and Success Rate (SR) of 55.09% on the challenging Bench2Drive benchmark. To the best of our knowledge, this is the first work to demonstrate the effectiveness of online reinforcement learning for the VLA model in autonomous driving.

URL PDF HTML 收藏
2601.00898 2026-07-20 cs.LG cs.RO 版本更新

Dichotomous Diffusion Policy Optimization

二元扩散策略优化

Ruiming Liang, Yinan Zheng, Kexin Zheng, Tianyi Tan, Jianxiong Li, Liyuan Mao, Zhihao Wang, Guang Chen, Hangjun Ye, Jingjing Liu, Jinqiao Wang, Xianyuan Zhan

机构 * Fundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(基础模型研究中心,自动化研究所,中国科学院) School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学) Institute for AI Industry Research (AIR), Tsinghua University(人工智能产业研究院(AIR),清华大学) The Chinese University of Hong Kong(香港中文大学) Shanghai Jiao Tong University(上海交通大学) Peking University(北京大学) Xiaomi EV(小米电动车)

AI总结 DIPOLE是一种新的RL算法,通过二元策略分解实现稳定可控的扩散策略优化,适用于复杂现实应用。

详情
AI中文摘要

基于扩散的策略在解决广泛决策任务中日益流行,因其在推理过程中具有优越的表达能力和可控的生成能力。然而,使用强化学习(RL)有效训练大型扩散策略仍然具有挑战性。现有方法要么由于直接最大化价值目标而面临训练不稳定的问题,要么由于依赖粗糙的高斯似然近似而面临计算问题,后者需要大量足够小的去噪步骤。在本工作中,我们提出了DIPOLE(二元扩散策略改进),一种新的RL算法,用于稳定且可控的扩散策略优化。我们首先回顾了RL中的KL正则化目标,该目标为扩散策略提取提供了有吸引力的加权回归目标,但通常难以在贪婪性和稳定性之间取得平衡。然后,我们制定了一种贪心化的策略正则化方案,这自然地使最优策略分解为一对稳定学习的二元策略:一个旨在最大化奖励,另一个专注于最小化奖励。在这样的设计下,优化的动作可以通过在推理过程中线性组合二元策略的得分来生成,从而实现对贪婪程度的灵活控制。在离线和离线到在线RL设置上对ExORL和OGBench的评估证明了我们方法的有效性。我们还使用DIPOLE训练了一个大型视觉-语言-动作(VLA)模型,用于端到端的自动驾驶(AD),并在大规模现实世界AD基准NAVSIM上进行评估,突显了其在复杂现实应用中的潜力。

英文摘要

Diffusion-based policies have gained growing popularity in solving a wide range of decision-making tasks due to their superior expressiveness and controllable generation during inference. However, effectively training large diffusion policies using reinforcement learning (RL) remains challenging. Existing methods either suffer from unstable training due to directly maximizing value objectives, or face computational issues due to relying on crude Gaussian likelihood approximation, which requires a large amount of sufficiently small denoising steps. In this work, we propose DIPOLE (Dichotomous diffusion Policy improvement), a novel RL algorithm designed for stable and controllable diffusion policy optimization. We begin by revisiting the KL-regularized objective in RL, which offers a desirable weighted regression objective for diffusion policy extraction, but often struggles to balance greediness and stability. We then formulate a greedified policy regularization scheme, which naturally enables decomposing the optimal policy into a pair of stably learned dichotomous policies: one aims at reward maximization, and the other focuses on reward minimization. Under such a design, optimized actions can be generated by linearly combining the scores of dichotomous policies during inference, thereby enabling flexible control over the level of greediness.Evaluations in offline and offline-to-online RL settings on ExORL and OGBench demonstrate the effectiveness of our approach. We also use DIPOLE to train a large vision-language-action (VLA) model for end-to-end autonomous driving (AD) and evaluate it on the large-scale real-world AD benchmark NAVSIM, highlighting its potential for complex real-world applications.

URL PDF HTML 收藏
2607.14727 2026-07-17 cs.CV 新提交

WorkDrive: Roadwork Chain of Causation for Autonomous Driving

WorkDrive:自动驾驶的道路施工因果链

Tianyi Jiang, Wen Zhang, Sihan Yang, Ming Lu, Wentao Zhang

机构 * Peking University(北京大学) Xiaomi EV(小米电动汽车)

AI总结 针对自动驾驶视觉语言模型在道路施工区域的难题,提出WorkDrive框架,通过自动化多任务感知管道提取场景事实,经监督微调与强化学习,在ROADWork数据集上降低轨迹平均位移误差,实现渐进改进。

详情
AI中文摘要

自动驾驶视觉语言模型(VLMs)在道路施工区域面临困难,熟悉的视觉线索改变或缺失,临时设备重新定义可行驶通道。VLMs能检测物体,但缺乏明确指导。我们提出WorkDrive框架,构建基于感知的施工区域因果推理并与轨迹预测对齐。自动化多任务感知管道提取结构化场景事实注入因果链注释管道,推理标签用于监督微调及强化学习。在最大公共施工区域数据集ROADWork上,所提道路施工因果链使轨迹平均位移误差(ADE)降低9.0%,基于一致性的GRPO再降低3.0%。代码和数据将公开发布。

英文摘要

Autonomous driving vision-language models (VLMs) struggle in roadwork zones, where familiar visual cues such as lane markings and permanent signs are altered or absent, and temporary devices such as cones and barriers redefine the drivable corridor. VLMs can detect these objects, but without explicit guidance they anchor their reasoning on familiar elements from pre-training and fail to connect work-zone observations to correct planning decisions. We propose WorkDrive, a framework that constructs perception-grounded causal reasoning for work zones and aligns it with trajectory prediction. An automated multitask perception pipeline extracts structured scene facts and injects them into a Chain-of-Causation (CoC) annotation pipeline, redirecting the annotator's attention to domain-specific elements. The resulting reasoning labels are used for supervised fine-tuning, followed by reinforcement learning with a single reward: consistency between lateral meta-actions and the predicted trajectory. On ROADWork, the largest public work-zone dataset, the proposed roadwork CoC reduces trajectory average displacement error (ADE) by 9.0\%, and consistency-based GRPO yields a further 3.0\%, achieving progressive improvement over the trajectory-only baseline. Code and data will be publicly released.

URL PDF HTML 收藏
2607.13712 2026-07-16 cs.CV cs.AI cs.CL cs.MM 新提交

Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs

Groc-PO:用于真实多模态大语言模型的基于上下文的偏好优化

Zhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, Zhendong Mao

机构 * University of Science and Technology of China(中国科学技术大学) Xiaomi Corporation(小米公司) State Key Laboratory of Communication Content Cognition, People’s Daily Online(人民日报社传播内容认知国家重点实验室)

AI总结 研究针对多模态大语言模型的不真实问题,提出基于上下文的偏好优化框架Groc-PO,构建相关数据集,通过多阶段偏好样本捕捉基础上下文,加强上下文相关推理,减轻跨阶段错误传播,提升模型性能。

Comments Accepted by ACM-MM 2026

详情
AI中文摘要

尽管多模态大语言模型取得了快速进展,但仍存在诸如视觉幻觉、内容编造和推理不准确等不真实问题,削弱了其可靠性和实用性。基于人类偏好的对齐方法如直接偏好优化(DPO)被广泛采用,但多模态推理错误常跨阶段传播,最终答案错误常源于早期基础阶段的错误,而标准DPO通常在最终答案层面进行偏好优化。为解决此问题,我们提出了用于多模态大语言模型的基于上下文的偏好优化框架Groc-PO。我们还构建了基于上下文的偏好数据集(GCPD),围绕对象基础、上下文基础和基于基础的推理三个阶段组织多阶段偏好样本,以捕捉基础上下文的形成、整合和利用。通过在多个基础阶段引入更明确的偏好监督,Groc-PO加强了上下文相关推理并减轻了跨阶段错误传播。大量实验表明,与标准DPO和其他强大基线相比,Groc-PO在减轻幻觉、忠实推理和整体可靠性方面取得了更好的性能,支持了更明确的基础监督对可信多模态推理的价值。

英文摘要

Despite the rapid progress of Multimodal Large Language Models (MLLMs), they still suffer from untruthfulness issues, such as visual hallucinations, content fabrication, and unfaithful reasoning, which substantially undermine their faithfulness and practical utility. Alignment methods based on human preference, such as Direct Preference Optimization (DPO), have been widely adopted to address these issues. However, multimodal reasoning errors often propagate across stages, and final-answer errors can often be traced to mistakes in early grounding stages, yet standard DPO typically applies preference optimization at the final-answer level. This credit-assignment challenge means that supervision for early grounding stages is indirect rather than stage-specific, making it difficult to suppress error propagation arising from grounding drift and context inconsistency. To address this, we propose Grounded Context Preference Optimization (Groc-PO), a grounded preference optimization framework for MLLMs. We further construct the Grounded Context Preference Dataset (GCPD), organizing multi-stage preference samples around three stages of Object Grounding, Contextual Grounding, and Grounded Reasoning, to capture the formation, integration, and utilization of grounded context. By introducing more explicit preference supervision over multiple grounded stages, Groc-PO strengthens context-dependent reasoning and mitigates cross-stage error propagation. Extensive experiments show that, compared with standard DPO and other strong baselines, Groc-PO achieves improved performance in hallucination mitigation, faithful reasoning, and overall reliability, supporting the value of more explicit grounded supervision for trustworthy multimodal reasoning.

URL PDF HTML 收藏
2607.13621 2026-07-16 cs.AI 新提交

UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following

UESF-Bench:统一的具身寻找与跟随的基准测试与探究

Kun Yu, Jianhua Yang, Yixiang Chen, Changwei Wang, Hongyuan Yu, Yan Huang, Fushuo Huo, Ya Jing, Zhumin Chen, Keji He

机构 * Shandong University(山东大学) Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所) Qilu University of Technology(齐鲁工业大学) Xiaomi Inc(小米公司) Beijing University Of Technology(北京工业大学) Hong Kong Polytechnic University(香港理工大学)

AI总结 研究针对现有具身智能体语言引导人类跟随基准测试的局限,引入UESF-Bench基准,提出SeekFollow-VLA框架,可处理语义引导探索等任务,实验显示该框架在单人和多人环境中比基线有明显改进,为统一具身寻找与跟随建立基线。

详情
AI中文摘要

语言引导的人类跟随是具身智能体的一项重要能力,但现有基准测试通常假设目标人物在情节开始时是可见的。这种设置简化了问题,忽略了一个更现实的要求:智能体通常需要先找到语言描述的目标,然后在动态环境中持续跟随该目标。近期工作虽已开始研究人类搜索,但现有设置通常在特定任务场景中评估,且往往依赖更强的环境先验知识。此外,它们通常将搜索和跟随视为 separate 任务,仍缺乏用于系统评估的统一基准。为解决这些限制,我们引入了统一的具身寻找与跟随基准测试(UESF-Bench),这是一个用于具身人类寻找与跟随的大规模多样化基准测试。该基准要求智能体处理语义引导的探索、可靠的行为切换和恢复以及延迟的身份定位。为此,我们提出了 SeekFollow-VLA,这是一个具有任务驱动路由机制的视觉-语言-行动框架,用于在寻找和跟随之间进行潜在阶段推理和转换建模。实验结果表明,SeekFollow-VLA 在单人和多人环境中均比单头和双头基线有明显改进,为统一的具身寻找与跟随建立了基线。

英文摘要

Language-guided human following is an important capability for embodied agents, but existing benchmarks typically assume that the target person is visible at the start of an episode. This setting simplifies the problem and overlooks a more realistic requirement: an agent often needs to first find a language-described target and then persistently follow that target in a dynamic environment. While recent work has started to study human search, existing settings are typically evaluated in task-specific scenarios and often rely on stronger prior knowledge of the environment. Moreover, they usually treat searching and following as separate tasks and still lack a unified benchmark for systematic evaluation. To address these limitations, we introduce the Unified Embodied Seeking and Following Benchmark (UESF-Bench), a large-scale and diverse benchmark for embodied human seeking and following. The benchmark requires agents to handle semantic-guided exploration, reliable behavior switching and recovery, and delayed identity grounding. To this end, we propose SeekFollow-VLA, a vision-language-action framework with a task-driven routing mechanism for latent phase inference and transition modeling between seeking and following. Experimental results show that SeekFollow-VLA achieves clear improvements over both single-head and dual-head baselines across single-person and multi-person environments, establishing a baseline for unified embodied seek-and-follow.

URL PDF HTML 收藏
2607.12111 2026-07-15 cs.LG cs.AI cs.DC cs.MA cs.NI 新提交

PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs

PFAdapter:用于个性化联邦多模态大语言模型的分层LoRA分解

Jing Liu, Kun Yang, Yan Wang, Dingkang Yang, Xiaoshuai Hao, Wei Zhang, Yang Liu, Wei Zhou

机构 * College of Future Information Technology, Fudan University(复旦大学未来信息技术学院) Division of Natural and Applied Sciences, Duke Kunshan University(昆山杜克大学自然科学与应用科学部) Department of Electrical and Computer Engineering, The University of British Columbia(英属哥伦比亚大学电气与计算机工程系) Ant Group(蚂蚁集团) College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院) School of Data Science and Engineering, East China Normal University(华东师范大学数据科学与工程学院) College of Intelligent Robotics and Advanced Manufacturing, Fudan University & Fysics AI(复旦大学智能机器人与先进制造学院及复肆智能科技(上海)有限公司) Xiaomi EV, Xiaomi Campus(小米汽车、小米园区) Information and Communications Technology Cluster, Singapore Institute of Technology (SIT)(新加坡理工学院信息通信技术集群) College of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院) School of Computer Science and Informatics, Cardiff University(卡迪夫大学计算机科学与信息学院)

AI总结 针对分布式网络中联邦微调平衡全局知识与局部适应的挑战,提出PFAdapter框架,用分层LoRA分解将适配器参数分离,通过正交正则化和选择性聚合协议减少通信成本,实验证明该框架在多数据集上性能优于基线,为智能AI部署提供有效方案。

Comments submitted to IEEE TCCN

详情
AI中文摘要

智能AI系统通过部署能协作学习并保持数据隐私的自主智能代理重塑通信和网络。在分布式网络环境中,多模态大语言模型是边缘设备的认知引擎,但联邦微调在异构网络条件下平衡全局知识聚合与局部适应方面面临挑战。传统联邦协议依赖统一参数聚合,导致个性化欠佳和通信开销大。为此提出PFAdapter框架,引入分层LoRA分解,将适配器参数分为全局共享和局部私有组件,通过正交正则化防止组件间冗余特征学习,选择性聚合协议仅同步全局共享组件,减少近50%通信成本。在多个数据集上实验表明,PFAdapter性能优于现有基线,在不同边缘智能任务上准确率提高2.4%至4.8%,为资源受限通信网络中的智能AI部署提供了有效解决方案。

英文摘要

Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents capable of collaborative learning while maintaining data privacy at network edges. Within distributed network environments, Multimodal Large Language Models (MLLMs) serve as cognitive engines for edge devices, yet federated fine-tuning faces substantial challenges in balancing global knowledge aggregation with local adaptation under heterogeneous network conditions. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain-invariant features with client-specific nuances, thereby resulting in suboptimal personalization and excessive communication overhead. To address these challenges, we propose PFAdapter, a communication-efficient framework introducing hierarchical LoRA decomposition to explicitly separate adapter parameters into global-shared and local-private components. Query and key projections are assigned to global synchronization for capturing universal multimodal semantics across the network, while value and output projections remain localized for edge-specific adaptation. Additionally, orthogonality regularization based on the Frobenius norm enforces strict separation between these components, preventing redundant feature learning. Selective aggregation protocols synchronize only global-shared components across the federated network, preserving local expertise and reducing communication costs by nearly 50%. Extensive experiments on VQA-RAD, SLAKE, Hateful Memes, and CrisisMMD datasets demonstrate that PFAdapter consistently outperforms state-of-the-art baselines, achieving accuracy improvements ranging from 2.4% to 4.8% across diverse edge intelligence tasks. Consequently, our framework establishes an efficient solution for agentic AI deployment in resource-constrained communication networks.

URL PDF HTML 收藏
2607.11643 2026-07-14 cs.RO cs.AI 新提交

Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model

小米机器人-U0:基于世界基础模型的统一具身合成

Xinghang Li, Jun Guo, Qiwei Li, Long Qian, Hang Lai, Yueze Wang, Hongyu Yan, Jiahang Cao, Xi Chen, Jingen Qu, Jiaxi Song, Nan Sun, Hanye Zhao, Futeng Liu, Wanli Peng, Heyun Wang, Yunhong Wang, Caoyu Xia, Jack Zhao, Diyun Xiang, Hangjun Ye, Heng Qu, Huaping Liu, Jason Li

机构 * Xiaomi Robotics(小米机器人)

AI总结 研究针对基础模型应用于具身场景受限问题,提出小米机器人-U0多模态自回归模型,将具身生成扩展为基础图像和视频生成的形式,联合优化多种生成任务,在多方面取得领先成果,证明基础世界模型可用于具身智能。

详情
AI中文摘要

近期的基础图像和视频生成模型具有强大的泛化和可控性,但直接应用于具身场景时受多视图一致性、几何连贯性和机器人实体约束等限制。现有方法通常用有限机器人数据适配基础模型,常牺牲大规模预训练获得的视觉知识。我们提出小米机器人-U0,一个380亿参数的多模态自回归模型用于统一具身合成。它将具身生成视为基础图像和视频生成的扩展,联合优化文本到图像生成、图像编辑、具身场景生成、具身转移和具身视频生成。该统一框架在适配具身设置时保留了预训练世界基础模型的泛化性。小米机器人-U0是首个支持跨多个机器人实体进行高质量多视图场景生成的模型,还引入结构化、可控的具身转移用于细粒度编辑,同时保持多视图一致性和交互动态。它在单步和序列生成任务上取得了领先成果,在具身场景生成和转移的人工评估中优于GPT-Image-2.0,在具身视频生成的世界竞技场中排名第一,在具有挑战性的真实世界操作任务中将pi_0.5的分布外成功率从36.9%提高到63.2%。这些结果表明基础世界模型可作为具身世界模型和具身智能的可扩展数据引擎。代码和检查点可在指定网址获取。

英文摘要

Recent foundation image and video generation models offer strong generalization and controllability, but their direct application to embodied scenarios is limited by requirements for multi-view consistency, geometric coherence, and robot embodiment constraints. Existing methods typically adapt foundation models with limited robot data, often sacrificing visual knowledge acquired during large-scale pre-training. We present Xiaomi-Robotics-U0, a 38-billion-parameter multimodal autoregressive model for unified embodied synthesis. It treats embodied generation as an extension of foundation image and video generation and jointly optimizes text-to-image generation, image editing, embodied scene generation, embodied transfer, and embodied video generation. This unified framework preserves the generalization of the pre-trained world foundation model while adapting it to embodied settings. Xiaomi-Robotics-U0 is the first model to support high-quality multi-view scene generation across multiple robot embodiments and to introduce structured, controllable embodied transfer for fine-grained editing while preserving multi-view consistency and interaction dynamics. It achieves state-of-the-art results on single-step and sequential generation tasks, outperforming GPT-Image-2.0 in human evaluations of embodied scene generation and transfer, ranking first on World Arena for embodied video generation, and improving the out-of-distribution success rate of pi_0.5 from 36.9% to 63.2% on challenging real-world manipulation tasks. These results show that foundation world models can serve both as embodied world models and scalable data engines for embodied intelligence. Code and checkpoints are available at https://robotics.xiaomi.com/xiaomi-robotics-u0.html.

URL PDF HTML 收藏
2607.11444 2026-07-14 cs.CL 新提交

UMoE:Unlocking Every Expert in Domain-Specific Training

UMoE:在特定领域训练中解锁每个专家

Xuefeng Li, Pengfei Liu

机构 * Qwen Team(通义千问团队) DeepSeek-AI(渊亭科技) Thinking Machines Lab(思维机器实验室) LLM-Core Xiaomi(小米大语言模型核心团队)

AI总结 研究针对特定领域训练中专家池不合理问题,提出UMoE流程,先修剪低显著性专家,再扩展专家池,最后应用标准SFT,该方法在多架构、多领域及多基准测试中优于直接SFT,能转化冗余容量,提升训练效果。

详情
AI中文摘要

混合专家(MoE)模型在不按比例增加计算成本的情况下扩展容量,已成为前沿大语言模型的关键架构。然而,特定领域的训练后继承了由混合领域预训练塑造的专家池:相当一部分专家在目标领域贡献很小,标准监督微调(SFT)使该专家池的组成保持不变。我们提出了一种简单的、保持预算的流程,在微调前将专家池重新调整到目标领域。给定一个目标领域,我们(1)修剪领域对齐显著性最低的专家,(2)通过基于扰动的专家扩展将专家池重新增长到原来的大小,(3)应用标准SFT。得到的模型保留了原来的专家数量、参数数量和推理成本。UMoE通过单一的冻结方法且无需每个领域的超参数调整,在两种MoE架构(Qwen3 - 30B - A3B和Qwen3.5 - 35B - A3B)、五个领域(数学、代码、科学、工具使用和智能编码)以及12个基准测试中始终优于直接SFT。在强大的内部数学语料库上,直接SFT已经超过Qwen3 - 30B - A3B - Thinking(82.81对81.06),而UMoE进一步将平均值提高到84.17,额外提高了1.36分,证明了对更强SFT机制的鲁棒性。数据缩放实验进一步表明,随着训练数据的增加,增益仍然存在。分析表明,直接SFT模型将大量路由专家计算分配给一个低显著性子集,该子集可以事后去除而平均降级很小;UMoE将这种冗余容量转化为有用的领域容量,并实现了更低的训练损失,在下游评估中所有难度级别都有增益。

英文摘要

Mixture-of-Experts (MoE) models scale capacity without proportional compute cost and have become a key architecture for frontier large language models (LLMs). Yet domain-specific post-training inherits an expert pool shaped by mixed-domain pre-training: a substantial subset of experts contributes little on the target domain, and standard supervised fine-tuning (SFT) leaves the composition of this pool unchanged. We propose a simple, budget-preserving pipeline that realigns the expert pool to the target domain before fine-tuning. Given a target domain, we (1) prune the experts with lowest domain-aligned saliency, (2) regrow the expert pool to its original size through perturbation-based expert expansion, and (3) apply standard SFT. The resulting model preserves the original expert count, parameter count, and inference cost. With a single frozen recipe and no per-domain hyperparameter tuning, UMoE consistently improves over direct sft across two MoE architectures (Qwen3-30B-A3B and Qwen3.5-35B-A3B), five domains (math, code, science, tool-use, and agentic coding), and 12 benchmarks. Representative improvements are 3.4 points in math average accuracy, 6.0 points on SWE-bench Verified. On a strong in-house math corpus, direct sft already surpasses Qwen3-30B-A3B-Thinking (82.81 vs.\ 81.06), yet UMoE further raises the average to 84.17, an additional 1.36 points, demonstrating robustness to a substantially stronger SFT regime. Data-scaling experiments further show that the gain persists as training data grows. Analysis reveals that the direct-SFT model allocates substantial routed-expert compute to a low-saliency subset that can be removed post hoc with little average degradation; UMoE turns this redundant capacity into useful domain capacity and achieves lower training loss, with gains spanning all difficulty levels in downstream evaluation.

URL PDF HTML 收藏
2607.11233 2026-07-14 cs.CV 新提交

Structure-Detail Decoupled Autoregressive Generation for Fast and High-Fidelity Virtual Try-On

用于快速和高保真虚拟试穿的结构-细节解耦自回归生成

Lu Yang, Xiaonan Hu, Yanan Li, Daqi Liu, Xiang Bai, Hao Lu

机构 * Huazhong University of Science and Technology(华中科技大学) Wuhan Institute of Technology(武汉工程大学) Xiaomi EV(小米汽车)

AI总结 研究虚拟试穿问题,提出STAR-VTON框架,通过解耦潜在空间结构合成与像素空间细节恢复,结合匹配信息细化器,实现高效高保真虚拟试穿,其中VAR-VTON效率高,像素空间细化器能有效恢复细节。

详情
AI中文摘要

虚拟试穿(VTON)是一个双条件图像生成问题,需要准确保留人物并忠实进行服装变形和细节合成。基于扩散的VTON方法在压缩潜在空间中联合建模这些因素,但因固有潜在压缩存在高频细节损失。近期视觉自回归(VAR)模型为高质量生成提供了有前景的替代方案,但因缺乏有效双条件机制未用于VTON。为此引入VAR-VTON,虽有效但在保留细粒度服装细节上仍有困难。进而提出STAR-VTON,通过解耦潜在空间结构合成与像素空间细节恢复,借助匹配信息细化器建立对应关系来恢复细节。实验表明STAR-VTON实现了效率与保真度的良好权衡,VAR-VTON比基于扩散的方法快至少4倍且不降低质量,像素空间细化器能有效恢复细节并可作为即插即用模块惠及现有VTON方法。

英文摘要

Virtual try-on (VTON) is a bi-conditional image generation problem that requires not only accurate person preservation but also faithful garment deformation and detail synthesis. Diffusion-based VTON methods can jointly model these factors in a compressed latent space, but suffer from high-frequency detail loss due to inherent latent compression, even with costly multi-step denoising. Recent visual autoregressive (VAR) models offer a promising alternative for high-quality generation with faster inference, yet remain unexplored for VTON due to the lack of effective bi-conditioning mechanisms. To bridge this gap, we first introduce VAR-VTON, a VAR-based VTON model that incorporates garment conditioning and structural guidance for efficient latent-space VTON. Despite its efficacy, latent-space generation still struggles to preserve fine-grained garment details. We argue that different VTON sub-tasks should be addressed in different representation spaces: structural synthesis such as garment warping and person layout is suited to the latent space, whereas fine-grained detail recovery should be tackled in the pixel space. Motivated by this insight, we further propose STAR-VTON, a Two-Stage AutoRegressive framework that builds upon VAR-VTON by decoupling latent-space structural synthesis from pixel-space detail recovery. Our idea is to resort to a matching-informed refiner to establish dense correspondences between the stage-one generation and the source garment to directly map fine-grained pixel-space details. Extensive experiments show that STAR-VTON achieves an impressive efficiency-fidelity trade-off: VAR-VTON runs at least $4\times$ faster than diffusion-based counterparts without degrading quality, and the pixel-space refiner effectively restores fine details and acts as a plug-and-play module that can benefit existing VTON approaches.

URL PDF HTML 收藏
2607.08572 2026-07-10 cs.CV 新提交

Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning

Switch-Reasoner:通过强化学习在多任务混合中学习何时思考

Yiyang Fang, Pei Fu, Jinjie Li, Jian Liang, Wenke Huang, Ruijie Luo, Shaojie Zhang, Jian Luan, Yi R. Fung, Mang Ye

机构 * Wuhan University(武汉大学) Xiaomi Inc(小米公司) Wuhan University of Technology(武汉理工大学) Nanyang Technological University(南洋理工大学) The Hong Kong University of Science and Technology(香港科技大学)

AI总结 研究针对多模态大语言模型在异构多任务中“先思考后回答”范式的低效及训练后学习何时思考不稳定的问题,提出基于GRPO的Switch-Reasoner框架,通过双层调节机制实现推理模式自适应选择,实验验证其能减少不必要推理并平衡准确率与效率。

详情
AI中文摘要

多模态大语言模型(MLLMs)通常遵循固定的“先思考后回答”范式,在异构多任务设置中效率低下,因为简单输入可能不需要显式推理,而困难输入则能从中大幅受益。在训练后学习何时思考也不稳定,不平衡的展开可能使模型倾向于总是思考或总是直接行为。我们提出了Switch-Reasoner,一个基于GRPO的框架,用于学习为MLLMs自适应选择推理模式。它将思考视为虚拟工具调用,允许模型直接回答或在回答前调用显式推理。为稳定此决策,我们引入了双层调节机制,平衡思考模式和直接模式的整体使用,并基于两种选择的相对收益提供样本级监督。在11个多模态任务上的实验表明,Switch-Reasoner减少了不必要的推理,同时保持了强大的性能,实现了更好的准确性-效率权衡。

英文摘要

Multimodal Large Language Models (MLLMs) often follow a fixed Think-then-Answer paradigm, which is inefficient in heterogeneous multitask settings because simple inputs may not require explicit reasoning while difficult ones can benefit substantially from it. Learning when to think is also unstable during post-training, where imbalanced rollouts can drive the model toward always-thinking or always-direct behavior. We propose Switch-Reasoner, a GRPO-based framework that learns to adaptively select reasoning modes for MLLMs. It treats thinking as a virtual tool invocation and allows the model to either answer directly or invoke explicit reasoning before answering. To stabilize this decision, we introduce a dual-level regulation mechanism that balances the overall use of Thinking Mode and Direct Mode while providing sample-level supervision based on the relative benefit of the two choices. Experiments on 11 multimodal tasks show that Switch-Reasoner reduces unnecessary reasoning while maintaining strong performance, achieving a better accuracy-efficiency trade-off.

URL PDF HTML 收藏
2607.08434 2026-07-10 cs.CV 新提交

DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models

DeltaV:在统一大型多模态模型中通过视觉状态更新进行思考

Pengjie Wang, Linger Deng, Zujia Zhang, Shaojie Zhang, Zhenbo Luo, Pei Fu, Jian Luan, Xiang Bai, Yuliang Liu

机构 * Huazhong University of Science and Technology(华中科技大学) Xiaomi Inc.(小米公司)

AI总结 研究针对统一大型多模态模型全图像生成范式的问题,提出DeltaV模型,通过视觉更新避免冗余,引入TSIM路由器匹配令牌预算,构建StructCoT数据集,实验证明该范式能减少视觉令牌、提升推理能力,DeltaV - 2B性能优异。

详情
AI中文摘要

当前统一大型多模态模型(ULMMs)通过文本推理和中间视觉状态支持交错多模态推理,但通常将每个视觉状态生成为完整图像。这种全图像生成范式引入了大量视觉令牌冗余,并稀释了对稀疏但推理关键状态转换的监督。我们提出了DeltaV,一种用视觉更新取代全图像生成的ULMM。基于历史视觉状态,DeltaV增量预测紧凑的更新令牌,以捕获推理步骤中的视觉变化,避免对不变内容的重复建模。为了使每次更新的令牌预算与视觉变化的幅度相匹配,DeltaV引入了时间相似性(TSIM)路由器,一旦边际重建增益低于阈值,就停止分配令牌。为了支持更多样化和可推广的推理,我们进一步构建了StructCoT,一个包含105万个样本、跨越44个任务领域的大规模交错多模态推理数据集。实验表明,视觉更新范式在不影响重建保真度的情况下,平均减少了55.6%的新生成视觉令牌,并比全图像生成提高了3.3%的多模态推理能力。在StructCoT和大规模多模态数据上训练的DeltaV - 2B在域内多模态推理评估中比更大的开源模型高出8.4%,在外部多模态推理和理解基准上比可比规模的Qwen3 - VL - 2B高出5.9%。代码、模型和StructCoT将在这个https URL上发布。

英文摘要

Current Unified Large Multimodal Models (ULMMs) support interleaved multimodal reasoning through textual reasoning and intermediate visual states, but typically generate each visual state as a full image. This full-image generation paradigm introduces substantial visual-token redundancy and dilutes supervision on sparse yet reasoning-critical state transitions. We propose DeltaV, a ULMM that replaces full-image generation with visual updates. Conditioned on historical visual states, DeltaV incrementally predicts compact update tokens that capture the visual changes across reasoning steps, avoiding repeated modeling of unchanged content. To align the token budget of each update with the magnitude of visual change, DeltaV introduces a temporal similarity (TSIM) Router, which stops allocating tokens once the marginal reconstruction gain falls below a threshold. To support more diverse and generalizable reasoning, we further construct StructCoT, a large-scale interleaved multimodal reasoning dataset with 1.05M samples spanning 44 task domains. Experiments show that the visual-update paradigm reduces newly generated visual tokens by 55.6\% on average without compromising reconstruction fidelity, and improves multimodal reasoning by 3.3\% over full-image generation. Trained with StructCoT and large-scale multimodal data, DeltaV-2B further outperforms substantially larger open-source models by 8.4\% on in-domain multimodal reasoning evaluations and surpasses the comparable-scale Qwen3-VL-2B by 5.9\% on external multimodal reasoning and understanding benchmarks. Code, models, and StructCoT will be released at https://github.com/Pengjie-W/DeltaV.

URL PDF HTML 收藏
2602.23802 2026-07-09 cs.AI cs.CV 版本更新

EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models

EMO-R3:多模态大语言模型中用于情感推理的反射强化学习

Yiyang Fang, Wenke Huang, Pei Fu, Yihao Yang, Kehua Su, Zhenbo Luo, Jian Luan, Mang Ye

机构 * School of Computer Science, Wuhan University(武汉大学计算机学院) MiLM Plus, Xiaomi Inc.(小米公司)

AI总结 针对多模态大语言模型情感推理难题,提出EMO-R3框架。引入结构化情感思维并设计反射情感奖励,经大量实验验证,该框架显著提升模型可解释性与情商,在多视觉情感理解基准测试中表现出色。

Comments Accepted by CVPR 2026

详情
AI中文摘要

多模态大语言模型在视觉推理和理解任务中取得了显著进展,但在捕捉人类情感的复杂性和主观性方面仍存在困难。基于监督微调的现有方法往往泛化能力有限且可解释性差,而诸如群体相对策略优化等强化学习方法与情感认知的内在特征不匹配。为应对这些挑战,我们提出了用于情感推理的反射强化学习(EMO-R3)框架,以增强多模态大语言模型的情感推理能力。具体而言,我们引入结构化情感思维,以指导模型以结构化和可解释的方式进行逐步情感推理,并设计了反射情感奖励,使模型能够基于视觉-文本一致性和情感连贯性重新评估其推理。大量实验表明,EMO-R3显著提高了多模态大语言模型的可解释性和情商,在多个视觉情感理解基准上取得了优异性能。

英文摘要

Multimodal Large Language Models (MLLMs) have shown remarkable progress in visual reasoning and understanding tasks but still struggle to capture the complexity and subjectivity of human emotions. Existing approaches based on supervised fine-tuning often suffer from limited generalization and poor interpretability, while reinforcement learning methods such as Group Relative Policy Optimization fail to align with the intrinsic characteristics of emotional cognition. To address these challenges, we propose Reflective Reinforcement Learning for Emotional Reasoning (EMO-R3), a framework designed to enhance the emotional reasoning ability of MLLMs. Specifically, we introduce Structured Emotional Thinking to guide the model to perform step-by-step emotional reasoning in a structured and interpretable manner, and design a Reflective Emotional Reward that enables the model to re-evaluate its reasoning based on visual-text consistency and emotional coherence. Extensive experiments demonstrate that EMO-R3 significantly improves both the interpretability and emotional intelligence of MLLMs, achieving superior performance across multiple visual emotional understanding benchmarks.

URL PDF HTML 收藏
2607.05133 2026-07-07 cs.CV 新提交

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

UNIVERSE:面向自动驾驶的基于灵活掩码调制模态生成的统一视频动作模型

Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye, Francesco Nex, Hao Cheng, Michael Ying Yang

机构 * University of Twente(特温特大学) Xiaomi EV(小米汽车) University of Cambridge(剑桥大学) University of Bath(巴斯大学)

AI总结 针对现有方案难以将视频建模收益迁移至轨迹生成的问题,提出单掩码调制扩散Transformer的统一模型,通过模态解耦掩码实现直接的视频监督,大幅提升跨域动作泛化与推理效率。

Comments 18 pages, 7 figures, 8 tables

详情
AI中文摘要

世界动作模型(WAM)通过将未来视频预测作为场景动力学与时序因果性的密集监督,在提升自动驾驶动作泛化能力方面展现出巨大潜力。但目前尚不清楚哪种架构能更好地将视频建模的收益迁移至轨迹生成任务。现有级联或双扩散Transformer(DiT)设计将视频想象与动作预测分离,削弱了视频学习到的世界动力学向轨迹分支的迁移能力:动作模型仍可能过拟合数据集特定的驾驶先验,而视频模型仅能间接对规划过程施加正则化约束。本文提出UNIVERSE,一款基于单掩码调制扩散Transformer构建的统一视频动作模型。通过在共享生成参数内联合训练未来视频潜变量与自车轨迹token,UNIVERSE可让密集视频监督直接引导轨迹去噪过程,实现更强的跨域动作泛化能力。为保证因果有效性与部署效率,本文引入模态解耦可见性掩码,该掩码可跨模态共享历史上下文,同时阻断未来视频与轨迹token之间的互注意力。该机制避免了未来目标泄露,且可在测试阶段移除未来视频去噪操作以实现纯轨迹推理,相较于联合视频动作推演实现4.3倍的速度提升,同时保持相当的规划精度。该模型同样支持纯视频推演与联合视频动作推演模式。实验结果表明,UNIVERSE在NAVSIM数据集上取得91.0的PDMS指标(双DiT变体仅为89.6),无需微调即可在nuScenes与Bench2Drive数据集上展现优异的零样本迁移性能,消融实验也验证了单DiT统一架构、视频联合训练与掩码模态解耦的核心作用。

英文摘要

World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. However, it remains unclear which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents future-target leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a $4.3\times$ speedup over joint video-action rollout while maintaining comparable planning accuracy. The same model also supports video-only and joint video-action rollouts. Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs. 89.6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.

URL PDF HTML 收藏
2607.04425 2026-07-07 cs.CL cs.AI cs.CV cs.LG cs.MM 新提交

UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning

UI-MOPD:用于持续GUI智能体学习的多平台策略蒸馏

Niu Lian, Alan Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Yaowei Wang, Shu-Tao Xia, Jinpeng Wang

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院) Xiaomi(小米) Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)) Zhejiang University(浙江大学) Peng Cheng Laboratory(鹏城实验室)

AI总结 针对构建多平台GUI智能体的挑战,构建高质量数据集Uni-GUI,提出UI-MOPD方法,通过多教师策略蒸馏实现持续学习,动态选教师并转移行为先验,实验证明其能平衡跨平台能力保留与新平台适应。

Comments Technical report. 25 pages, 5 figures, 7 tables

详情
AI中文摘要

多模态基础模型和智能体系统的进展推动GUI智能体从单平台任务执行向跨平台交互发展。构建多平台GUI智能体仍具挑战。一方面,高质量可执行跨平台交互轨迹稀缺,现有数据平台覆盖有限;另一方面,不同平台交互惯例不同,联合或持续训练易出现问题。为应对这些挑战,我们构建了Uni-GUI,一个高质量的跨平台GUI交互数据集,并提出了UI-MOPD,这是第一种将多教师策略蒸馏纳入GUI智能体持续学习的方法。UI-MOPD根据当前环境动态选择特定平台的教师,并通过平台条件蒸馏将特定平台的行为先验转移到共享策略中,从而在保留现有平台能力的同时适应新平台。在OSWorld和MobileWorld上的实验表明,UI-MOPD分别实现了38.2%和12.0%的任务成功率,证明了其在平衡跨平台能力保留和新平台适应方面的有效性。项目页面:此https URL。

英文摘要

Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction. However, building multi-platform GUI agents remains challenging. On one hand, high-quality and executable cross-platform interaction trajectories are still scarce, and existing data often suffer from limited platform coverage. On the other hand, different platforms exhibit distinct interaction conventions, making joint or continual training prone to behavioral pattern mixing, platform-specific capability degradation, and catastrophic forgetting. To address these challenges, we construct Uni-GUI, a high-quality cross-platform GUI interaction dataset, and propose UI-MOPD, the first method that incorporates multi-teacher on-policy distillation into continual learning for GUI agents. UI-MOPD dynamically selects a platform-specific teacher according to the current environment and transfers platform-specific behavioral priors to a shared policy through platform-conditioned distillation, enabling adaptation to new platforms while preserving capabilities on existing ones. Experiments on OSWorld and MobileWorld show that UI-MOPD achieves task success rates of 38.2% and 12.0%, respectively, demonstrating its effectiveness in balancing cross-platform capability retention and new-platform adaptation. Project page: https://elispectre.github.io/UI-MOPD/.

URL PDF HTML 收藏
2606.20280 2026-07-07 cs.IR cs.AI 新提交

ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

ELVA:探索排序驱动的通用多模态检索

Yuhan Liu, Pei Fu, Hang Li, Yukun Qi, Chao Jiang, Jingwen Fu, Zhen Liu, Bin Qin, Zhenbo Luo, Jian Luan, Jingmin Xin

机构 * National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家级重点实验室) Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院) MiLM Plus Xiaomi Inc(小米公司) Zhongguancun Academy(中关村学院) Beijing, China(北京市)

AI总结 提出ELVA框架,通过基于规则的强化学习缓解对比学习中的粒度盲视问题,在通用多模态检索中实现排序优化,并在新基准MRBench上提升13.1%。

Comments Accepted by ECCV 2026

详情
AI中文摘要

利用多模态大语言模型(MLLMs)进行对比学习已成为提升通用多模态检索(UMR)性能的主流范式。然而,先前的工作在将对比范式适应到检索任务时忽略了粒度盲视问题。粒度盲视是指模型倾向于忽略查询中包含的粒度级信息,而这些信息对于有效处理复杂查询至关重要。这源于对比学习将样本视为二元分类(正/负),而忽略了每个负样本携带的不同信息。为了解决这个问题,我们认为应该根据负样本与正样本的相似度区别对待它们,使模型能够从每个负样本中学习不同的粒度信息。在本文中,我们引入了一个简单但有效的框架,称为ELVA,一种新颖的基于规则的强化学习框架,通过排序驱动的MLLMs缓解粒度盲视。1)不依赖奖励模型,我们将可验证奖励的强化学习(RLVR)扩展到检索任务,使模型能够探索新的排序行为而无需显式的排序标签。2)通过利用基于规则的奖励,我们的方法联合优化负样本的排序,同时扩大正负样本之间的相似度差距。为了更精确地衡量粒度盲视,我们进一步引入了MRBench,一个专门为多粒度查询场景设计的新基准。ELVA在标准检索基准上取得了最先进的结果,在MRBench上显著提升13.1%,进一步证明了其在缓解粒度盲视方面的有效性。

英文摘要

Leveraging Multimodal Large Language Models (MLLMs) via contrastive learning has become a mainstream paradigm for improving the performance of Universal Multimodal Retrieval (UMR). However, previous works have ignored the grain blindness when adapting the contrastive paradigm into retrieval tasks. Grain blindness refers to the tendency of the model to overlook grain-level information contained in the query, which is crucial for effectively handling complex queries. This stems from contrastive learning treating samples as a binary classification (positive/negative), while ignoring the different information carried by each negative sample. To address this, we argue that negatives should be treated differently according to their similarity to the positive sample, enabling the model to learn distinct grain information from each negative. In this paper, we introduce a simple but effective framework, called ELVA, a novel rule-based RL framework that mitigates grain blindness through ranking-driven MLLMs. 1) Instead of relying on reward models, we extend Reinforcement Learning with Verifiable Rewards (RLVR) to retrieval tasks, allowing the model to explore new ranking behaviors without explicit ranking labels. 2) By utilizing rule-based rewards, our approach jointly optimizes the ranking of negative samples while enlarging the similarity gap between positive and negative. To more precisely measure grain blindness, we further introduce MRBench, a new benchmark specifically designed for multi-grain query scenarios. ELVA achieves state-of-the-art results across standard retrieval benchmarks, and its notable 13.1% improvement on MRBench further demonstrates its effectiveness in alleviating grain blindness.

URL PDF HTML 收藏
2603.03283 2026-07-07 cs.CV 版本更新

Utonia: Toward One Encoder for All Point Clouds

Utonia:迈向适用于所有点云的单一编码器

Yujia Zhang, Xiaoyang Wu, Yunhan Yang, Xianzhe Fan, Han Li, Yuechen Zhang, Zehao Huang, Naiyan Wang, Hengshuang Zhao

机构 * The University of Hong Kong(香港大学) The Chinese University of Hong Kong(香港中文大学) Xiaomi(小米)

AI总结 研究旨在训练跨多领域的单自监督点变压器编码器Utonia,其能学习一致表征空间实现跨域转移,统一各领域点云,提升感知能力,还对具身和多模态推理有益,有望成为稀疏3D数据基础模型。

Comments produced by Pointcept, project page: https://pointcept.github.io/Utonia

Journal ref International Conference on Machine Learning 2026

详情
AI中文摘要

我们憧憬一个未来,所有领域的点云能共同塑造一个使各方受益的单一模型。为此,我们提出Utonia,朝着在跨不同领域(包括遥感、户外激光雷达、室内RGB-D序列、以物体为中心的CAD模型以及从仅RGB视频提取的点云)训练单个自监督点变压器编码器迈出了第一步。尽管它们具有不同的传感几何形状、密度和先验知识,但Utonia学习了一个跨域转移的一致表征空间。这种统一提高了感知能力,同时揭示了只有在联合训练域时才会出现的有趣的新兴行为。除了感知,我们观察到Utonia表征还可以使具身和多模态推理受益:以Utonia特征为视觉语言动作策略提供条件可以改善机器人操作,将它们集成到视觉语言模型中可以在空间推理方面取得进展。我们希望Utonia可以作为迈向稀疏3D数据基础模型的一步,并支持AR/VR、机器人技术和自动驾驶中的下游应用。

英文摘要

We dream of a future where point clouds from all domains can come together to shape a single model that benefits them all. Toward this goal, we present Utonia, a first step toward training a single self-supervised point transformer encoder across diverse domains, spanning remote sensing, outdoor LiDAR, indoor RGB-D sequences, object-centric CAD models, and point clouds lifted from RGB-only videos. Despite their distinct sensing geometries, densities, and priors, Utonia learns a consistent representation space that transfers across domains. This unification improves perception capability while revealing intriguing emergent behaviors that arise only when domains are trained jointly. Beyond perception, we observe that Utonia representations can also benefit embodied and multimodal reasoning: conditioning vision-language-action policies on Utonia features improves robotic manipulation, and integrating them into vision-language models yields gains on spatial reasoning. We hope Utonia can serve as a step toward foundation models for sparse 3D data, and support downstream applications in AR/VR, robotics, and autonomous driving.

URL PDF HTML 收藏
2606.05778 2026-07-03 cs.CV 版本更新

Beyond Absolute Scores: Relative Edit-induced Difference for Generalizable Image Aesthetic Assessment

超越绝对分数:基于编辑诱导差异的通用图像美学评估

Qifei Jia, Xintong Yao, Yasen Zhang, Minghao Li, Yajie Chai, Qiming Lu, Baoyue Shen, Runyu Shi, Ying Huang, Yue Zhang

机构 * Xiaomi Corporation, Beijing, China(小米公司,北京,中国)

AI总结 提出RED-Aes框架,利用可控图像编辑模型模拟人类审美推理,通过相对编辑诱导差异学习通用美学原则,实现跨场景泛化。

详情
AI中文摘要

传统的图像美学评估(IAA)方法主要依赖于回归绝对平均意见分数(MOS)。然而,这种范式忽视了人类审美感知固有的动态性质,这种感知依赖于对隐含视觉参考的无意识比较。因此,缺乏对美学差异的因果推理使得模型无法学习通用的美学原则,从而限制了它们在多样化场景中的泛化能力。在这项工作中,我们重新思考IAA任务,并提出相对编辑诱导差异美学学习(RED-Aes),一种新颖的框架,利用可控图像编辑模型模拟人类审美推理过程。RED-Aes不拟合绝对分数分布,而是显式学习驱动美学变化的视觉因素。为了支持这一范式,我们构建了RED-20k数据集,包含基于编辑的图像对、定量美学差异和思维链(CoT)推理。此外,我们引入了一种由相对排序一致性奖励引导的三阶段训练策略,仅通过相对监督优化模型。大量实验表明,RED-Aes在多个公共基准上取得了最先进的性能,展现出优越的泛化能力。

英文摘要

Traditional Image Aesthetic Assessment (IAA) methods mainly rely on regressing absolute Mean Opinion Scores (MOS). However, such a paradigm overlooks the inherently dynamic nature of human aesthetic perception, which relies on subconscious comparison against implicit visual references. Consequently, the lack of causal reasoning regarding aesthetic differences prevents models from learning generalizable aesthetic principles, thus limiting their generalization across diverse scenarios. In this work, we rethink the IAA task and propose Relative Edit-induced Difference Aesthetic learning (RED-Aes), a novel framework that leverages controllable image editing models to simulate the human aesthetic reasoning process. Instead of fitting absolute score distributions, RED-Aes explicitly learns the visual factors that drive aesthetic changes. To support this paradigm, we construct the RED-20k dataset, which comprises editing-based image pairs, quantitative aesthetic differences, and Chain-of-Thought (CoT) reasoning. Furthermore, we introduce a three-stage training strategy guided by a relative ranking consistency reward, optimizing the model solely via relative supervision. Extensive experiments demonstrate that RED-Aes achieves state-of-the-art performance on multiple public benchmarks, exhibiting superior generalization capabilities.

URL PDF HTML 收藏
2606.05575 2026-07-03 cs.SD eess.AS 版本更新

SB-RF: Schrödinger Bridge Rectified Flow for One-Step Robust Speech Enhancement

SB-RF: 用于一步鲁棒语音增强的薛定谔桥整流流

Caixia Lu, Xueyang Lv, Penglong Hu, Jiaming Xu

机构 * Xiaomi Corporation, Beijing, China(小米公司,北京,中国)

AI总结 提出SB-RF,一种结合整流流与薛定谔桥理论的一步生成式语音增强框架,通过熵正则化最优传输构建条件桥,实现高质量一步生成,在VoiceBank-DEMAND基准上达到领先性能,并在低信噪比场景下展现出强鲁棒性和高效率。

详情
AI中文摘要

生成模型在语音增强中表现出令人印象深刻的结果,但通常受限于多步推理。我们提出SB-RF,一种将整流流(RF)与薛定谔桥(SB)理论相结合的一步生成框架。SB-RF通过熵正则化最优传输在干净和带噪语音分布之间构建条件桥。通过RF的速度匹配目标将SB轨迹与最优传输测地线对齐,SB-RF能够通过一步生成实现高质量增强。实验表明,SB-RF在VoiceBank-DEMAND基准上达到了生成方法中的领先性能。此外,为了全面评估在具有挑战性的真实场景中的性能,我们在一个模拟的低信噪比测试集上使用扩大的训练数据集评估SB-RF。在这些条件下,SB-RF展现出强大且具有竞争力的鲁棒性和高效率,验证了其在现实应用中的潜力。

英文摘要

Generative models have shown promising results for speech enhancement (SE), but they often rely on multi-step inference, limiting low-latency deployment. We propose SB-RF, a one-step generative framework that integrates Rectified Flow (RF) with Schrödinger Bridge (SB) theory. During training, SB-RF samples intermediate states from an SB time marginal and trains a conditional velocity field with the RF velocity-matching objective. At inference, SB-RF starts from the noisy observation and applies a single Euler update. Experiments show that SB-RF achieves competitive performance among generative methods on the VoiceBank-DEMAND benchmark. To further assess performance beyond this standard setting, we evaluate SB-RF on a simulated low signal-to-noise ratio test set using an expanded training dataset. Under these conditions, SB-RF achieves superior performance over the compared baselines, supporting its potential for real-world applications.

URL PDF HTML 收藏
2607.00498 2026-07-02 cs.CV 新提交

Robust 3D Alignment of Generative Reconstructions via Partial Monocular Observations

基于部分单目观测的生成式重建鲁棒3D对齐

Yuchen Zhang, Luanyuan Dai, Yiwei Wang, Xiwei Xu, Jianing Zhang, Johnny. r. zhang, Xianhui Meng, Yanbiao Ma, Jiayi Ma, Xiaoshuai Hao

机构 * School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学与技术学院) School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院) School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院) School of Tech Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院) Independent Researcher(独立研究员) School of Electronic Engineering and Information Science, University of Science and Technology of China(中国科学技术大学电子工程与信息科学学院) Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院) School of Robotics, Wuhan University(武汉大学机器人学院) Xiaomi EV(小米汽车)

AI总结 针对生成式3D重建与部分单目观测对齐的挑战,提出无训练可解释几何对齐框架,通过Sim(3)变换恢复度量尺度与位姿,引入幻觉滤波抑制异常,在基准测试中显著优于传统和学习方法。

详情
AI中文摘要

将生成式3D重建与部分单目观测对齐是计算机视觉中一个关键但探索不足的挑战。由于噪声、稀疏的单目输入与密集生成先验之间存在严重不对称性,其尺度模糊和几何幻觉,加上缺乏初始重叠,使得传统配准流程失效。为解决这些问题,我们提出一个无需训练且可解释的几何对齐框架,通过3D相似变换(Sim(3))锚定生成式3D先验,从而恢复准确的度量尺度和位姿。具体而言,我们引入显式尺度因子以解决度量模糊性,并采用粗到细的对齐策略,利用几何感知描述子进行鲁棒初始化,以及解耦闭式求解器进行精度细化。此外,我们引入幻觉滤波操作以有效抑制由幻觉几何引起的异常。为了在这些极端条件下评估对齐性能,我们提出了GenPMOAlign--Where2Place,一个专门为生成式到部分单目观测对齐设计的严格基准。实验表明,我们的方法实现了稳定且准确的配准,显著优于经典几何流程和最新的基于学习的基线。代码和基准将公开发布。

英文摘要

Aligning generative 3D reconstructions with partial monocular observations is a critical but under-explored challenge in computer vision. This task is inherently ill-posed due to severe asymmetries between noisy, sparse monocular inputs and dense generative priors, whose scale ambiguity and geometric hallucinations, combined with the lack of initial overlap, render traditional registration pipelines ineffective. To resolve these issues, we propose a training-free and interpretable geometric alignment framework that grounds generative 3D priors via a 3D similarity transformation (Sim(3)), which can recover accurate metric scale and pose. Specifically, we introduce an explicit scale factor to resolve metric ambiguity and employ a coarse-to-fine alignment strategy, leveraging geometry-aware descriptors for robust initialization and a decoupled closed-form solver for precision refinement. In addition, we introduce a Hallucination Filtering operation to effectively suppress outliers caused by hallucinated geometry. To evaluate alignment performance under these extreme conditions, we introduce GenPMOAlign--Where2Place, a rigorous benchmark specifically designed for Generative-to-Partial Monocular Observational Alignment. Experiments demonstrate that our method achieves stable and accurate registration, substantially outperforming both classical geometric pipelines and state-of-the-art learning-based baselines. Code and the benchmark will be publicly released.

URL PDF HTML 收藏
2604.04198 2026-07-02 cs.CV cs.RO 版本更新

DriveVA: Video Action Models are Zero-Shot Drivers

DriveVA: 视频动作模型是零样本驱动器

Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye, Michael Ying Yang, Francesco Nex, Hao Cheng

机构 * University of Twente(特温特大学) Xiaomi EV(小米电动汽车) University of Cambridge(剑桥大学) University of Bath(巴斯大学)

AI总结 DriveVA提出了一种新的自动驾驶世界模型,通过共享潜在生成过程联合解码未来视觉预测和动作序列,提升跨数据集和传感器配置的泛化能力,减少碰撞率和误差。

Comments Accepted to ECCV 2026. 30 pages, 12 figures, 11 tables

详情
AI中文摘要

通用性是自动驾驶中的核心挑战,因为现实部署需要在未见场景、传感器领域和环境条件下保持稳健性能。最近基于世界模型的规划方法在场景理解和多模态未来预测方面表现出色,但其跨数据集和传感器配置的泛化能力仍有限。此外,其松散耦合的规划范式常导致视觉想象中的视频轨迹不一致。为克服这些限制,我们提出了DriveVA,一种新的自动驾驶世界模型,通过共享潜在生成过程联合解码未来视觉预测和动作序列。DriveVA继承了大规模视频生成模型中对运动动态和物理合理性丰富的先验知识,以捕捉连续的时空演变和因果交互模式。为此,DriveVA采用基于DiT的解码器,联合预测未来动作序列(轨迹)和视频,使规划与场景演变更紧密对齐。我们还引入了视频延续策略以加强长时间滚动一致性。DriveVA在挑战NAVSIM上实现了90.9 PDM分数的 impressive closed-loop 性能。广泛的实验也展示了DriveVA的零样本能力和跨域泛化能力,与最先进的基于世界模型的规划器相比,其在nuScenes上平均L2误差和碰撞率分别减少了78.9%和83.3%,在基于CARLA v2的Bench2drive上分别减少了52.5%和52.4%。

英文摘要

Generalization is a central challenge in autonomous driving, as real-world deployment requires robust performance under unseen scenarios, sensor domains, and environmental conditions. Recent world-model-based planning methods have shown strong capabilities in scene understanding and multi-modal future prediction, yet their generalization across datasets and sensor configurations remains limited. In addition, their loosely coupled planning paradigm often leads to poor video-trajectory consistency during visual imagination. To overcome these limitations, we propose DriveVA, a novel autonomous driving world model that jointly decodes future visual forecasts and action sequences in a shared latent generative process. DriveVA inherits rich priors on motion dynamics and physical plausibility from well-pretrained large-scale video generation models to capture continuous spatiotemporal evolution and causal interaction patterns. To this end, DriveVA employs a DiT-based decoder to jointly predict future action sequences (trajectories) and videos, enabling tighter alignment between planning and scene evolution. We also introduce a video continuation strategy to strengthen long-duration rollout consistency. DriveVA achieves an impressive PDM-based planning performance of 90.9 PDM score on the NAVSIM benchmark. Extensive experiments also demonstrate the zero-shot capability and cross-domain generalization of DriveVA, which reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes and 52.5% and 52.4% on the Bench2Drive built on CARLA v2 compared with the state-of-the-art world-model-based planner.

URL PDF HTML 收藏
2606.24336 2026-07-01 cs.CV 新提交

TIGER: Taming Identity, Geometry, and Generative Priors for High-Quality Face Video Restoration

TIGER:驯服身份、几何和生成先验以实现高质量人脸视频恢复

Yang Zhou, Wenxue Li, Peng Zhang, Yifei Chen, Fei Wang, Daiguo Zhou

机构 * MiLM Plus, Xiaomi Inc.(小米公司MiLM Plus) The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

AI总结 提出TIGER框架,融合身份、几何和生成先验,通过解耦3D参数空间和一步整流流,解决人脸视频恢复中的身份偏移、视角纠缠和感知真实性问题,实现高保真、时间一致且高效的恢复。

详情
AI中文摘要

人脸视频恢复(FVR)旨在从退化的输入中恢复高保真的人脸视频,同时保持跨帧的身份和语义一致性。现有方法通常难以同时解决三个关键挑战:身份偏移、视角纠缠的指导和感知真实性。为了解决这些问题,我们提出了TIGER,一个结构化的三先验融合框架,它驯服了身份、几何和生成先验以实现高质量的FVR。具体来说,首先通过将主题判别性嵌入注入潜在空间来建立身份先验,有效地将主题的身份锚定在严重退化下。然后,为了为动态视频提供时间一致的结构指导,TIGER通过将2D参考线索提升到解耦的3D参数空间来构建几何先验,通过跨源参数融合创建几何锚点。此外,为了在不牺牲真实性的情况下实现最大效率,我们通过一步整流流利用视频生成模型的生成先验。我们进一步设计了一种渐进的三阶段训练优化策略,该策略细化结构保真度、纹理重建和分布级真实性,以确保稳健优化。我们还构建了一个大规模FVR数据集,以促进稳健训练和标准化评估。大量实验表明,TIGER在身份保真度和时间稳定性方面均达到了最先进的性能,实现了高质量、高效且身份一致的FVR。项目页面:此https URL。

英文摘要

Face Video Restoration (FVR) aims to recover high-fidelity facial videos from degraded input while preserving identity and semantic consistency across frames. Existing methods often struggle to simultaneously address three key challenges: identity shift, viewpoint-entangled guidance, and perceptual realism. To tackle these issues, we propose TIGER, a structured tri-prior fusion framework that Tames Identity, Geometry, and gEnerative pRiors for high-quality FVR. Specifically, an Identity Prior is first established by injecting subject-discriminative embeddings into the latent space, effectively anchoring the subject's identity against severe degradations. Then, to provide temporally consistent structural guidance for dynamic videos, TIGER constructs a Geometry Prior by lifting 2D reference cues into a disentangled 3D parameter space, creating a geometric anchor through cross-source parameter fusion. Moreover, to achieve maximum efficiency without compromising realism, we harness the video generation model's Generative Prior through a one-step rectified flow. We further design a progressive three-stage training optimization strategy that refines structural fidelity, textural reconstruction, and distribution-level realism to ensure robust optimization. We also construct a large-scale FVR dataset to facilitate robust training and standardized evaluation. Extensive experiments demonstrate that TIGER achieves state-of-the-art performance in both identity fidelity and temporal stability, delivering a high-quality, efficient and identity-consistent FVR. Project page: https://yzhoulv.github.io/Tiger/.

URL PDF HTML 收藏
2605.04057 2026-07-01 cs.LG cs.AI 版本更新

Structured Progressive Knowledge Activation for LLM-Driven Neural Architecture Search

结构化渐进知识激活用于LLM驱动的神经架构搜索

Zhen Liu, Yuhan Liu, Jinjun Wang, Wei Song, Jianyi Liu, Jingwen Fu

机构 * State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University.(人机混合增强智能国家重点实验室,人工智能与机器人研究院,西安交通大学) MiLM Plus, Xiaomi Inc.(小米MiLM Plus) North China University of Technology, Beijing.(华北电力大学(北京)) Zhongguancun Academy, Beijing, China(中关村学院,北京,中国)

AI总结 本文提出SPARK方法,通过显式选择功能因素并条件化编辑,减少功能耦合影响,提升神经架构搜索的样本效率和OOD准确性。

详情
AI中文摘要

本文聚焦神经架构搜索(NAS)中的关键挑战:在昂贵评估下整合已知架构知识并探索新设计。大语言模型(LLMs)作为NAS的有力助手,能将丰富的架构和编码先验转化为可执行代码修改。然而,实践中看似局部的修改常导致非局部行为和性能变化,因单个修改可能意外耦合多个交互功能因素,即功能耦合。为使LLM知识在耦合下有效,我们提出结构化渐进知识激活(SPARK),通过显式选择功能因素并条件化编辑,减少耦合副作用,产生更精确可靠的架构修改。在CLRS-DFS上,SPARK实现了28.1倍的样本高效架构进化加速,并在OOD准确性上获得22.9%的相对提升。

英文摘要

This paper focuses on a key challenge in Neural Architecture Search (NAS): integrating established architectural knowledge while exploring new designs under expensive evaluations. Large language models (LLMs) are a promising assistant for NAS because they can translate rich architectural and coding priors into executable code edits. However, in practice, seemingly local revisions often propagate into non-local behavioral and performance shifts because a single edit can inadvertently couple multiple interacting functional factors, a phenomenon we refer to as functional entanglement. To make LLM knowledge usable under such entanglement, we propose Structured Progressive Knowledge Activation (SPARK), which activates relevant priors by explicitly selecting the functional factor to modify and conditioning the edit on that factor. This factor-conditioned editing reduces entangled side effects and yields more targeted, reliable architecture modifications. On CLRS-DFS, SPARK achieves a 28.1x sample-efficient architecture evolution speedup and yields a 22.9\% relative improvement in OOD accuracy. Our code is available at https://github.com/AIM-ResearchLab/SPARK.

URL PDF HTML 收藏
2509.15443 2026-07-01 cs.RO cs.AI 版本更新

A Scalable Whole-body Motion Transfer via Implicit Kinodynamic Motion Retargeting

可扩展的全身运动迁移:基于隐式动力学运动重定向

Xingyu Chen, Hanyu Wu, Sikai Wu, Mingliang Zhou, Diyun Xiang, Haodong Zhang, Yangchen Zhou, Yukang Gao, Yi Gu, Renjing Xu

机构 * HKUST(GZ)(香港科技大学(广州)) ETH Zurich(苏黎世联邦理工学院) Zhejiang University(浙江大学) Xiaomi Robotics Lab(小米机器人实验室)

AI总结 提出隐式动力学运动重定向(IKMR),利用骨架图卷积双自编码器将人体与机器人运动映射到共享拓扑潜空间,结合物理信息精炼实现高速、鲁棒的运动数据转换,解决噪声与计算瓶颈。

Comments RSS 2026 Workshop. Webpage: https://cybercal.github.io/webpage.ikmr

详情
AI中文摘要

人形机器人模仿学习通过利用大规模人体运动数据,为解决机器人领域严重的数据稀缺瓶颈提供了有前景的途径。然而,扩展这一范式需要应对两个关键挑战。首先,从视频、动作捕捉系统或生成模型获取的人体运动数据通常包含空间噪声、抖动和帧级闪烁,这些在重定向过程中可能被放大,导致不安全或物理上不可行的机器人运动。其次,现有的运动重定向方法通常依赖逐帧数值优化,计算成本过高,难以用于大规模数据集合成。为克服这些限制,我们提出隐式动力学运动重定向(IKMR),一种高度可扩展的基于神经的数据转换流水线。IKMR利用基于骨架的图卷积双自编码器,将跨结构的人体与人形机器人运动学配置映射到共享拓扑潜空间。为保证生成数据的物理可行性,该框架包含一个物理信息精炼阶段,利用模拟的物理跟踪反馈学习鲁棒的运动先验。这种隐式公式从根本上解决了两个挑战。通过将计算负担从在线优化转移到离线推理,IKMR实现了超过每秒5000帧的前所未有的数据转换吞吐量。此外,利用学习到的运动先验,它作为内在的数据筛选机制,自然滤除源数据中的高频噪声和空间抖动,生成确保物理硬件安全的平滑轨迹。包括在人形机器人上部署全身控制在内的广泛评估证实,IKMR弥合了人体运动与机器人数据之间的鸿沟。

英文摘要

Human-to-humanoid imitation learning presents a promising pathway to address the severe data scarcity bottleneck in robotics by utilizing abundant, large-scale human motion collections. However, scaling this paradigm requires addressing two key challenges. First, human motion data acquired from videos, motion capture systems, or generative models often contains spatial noise, jitter, and frame-level flickering, which can be amplified during retargeting and lead to unsafe or physically infeasible robot motions. Second, existing motion retargeting methods typically rely on frame-by-frame numerical optimization, making them too computationally expensive for large-scale dataset synthesis. To overcome these limitations, we introduce Implicit Kinodynamic Motion Retargeting (IKMR), a highly scalable, neural-based data transformation pipeline. IKMR leverages a skeleton-based graph convolutional dual autoencoder to map cross-structural human and humanoid kinematic configurations into a shared topological latent space. To guarantee the physical viability of the generated data, the framework incorporates a physics-informed refinement phase that utilizes simulated physical tracking feedback to learn a robust motion prior. This implicit formulation fundamentally resolves both challenges. By shifting the computational burden from online optimization to offline inference, IKMR achieves an unprecedented data conversion throughput exceeding 5000 frames per second. Furthermore, leveraging the learned motion prior, it functions as an intrinsic data curation mechanism and naturally filters out high-frequency noise and spatial jitters from source data, yielding smooth trajectories that ensure physical hardware safety. Extensive evaluations, including real-world whole-body control deployments on humanoid robot, confirm that IKMR bridges the gap between human motion and robotic data.

URL PDF HTML 收藏
2606.30406 2026-06-30 cs.CL cs.LG

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

MOPD: 面向LLM后训练中能力集成的多教师同策略蒸馏

Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, Fuli Luo

机构 * Peking University(北京大学) LLM Core Xiaomi(小米大模型核心团队) University of Hong Kong(香港大学) Renmin University of China(中国人民大学)

AI总结 提出多教师同策略蒸馏(MOPD)范式,通过先训练领域专家再在学生自生成数据上蒸馏,消除暴露偏差并提供密集优化信号,在Qwen3-30B-A3B上优于多种基线,并已部署于工业级模型MiMo-V2-Flash。

详情
AI中文摘要

现代大型语言模型(LLMs)在后训练中依赖强化学习来提升特定能力,但将多种能力集成到一个模型中仍然困难。现有方法如Off-Policy Finetune和Mix-RL要么效率低下,要么性能损失。在这项工作中,我们提出了多教师同策略蒸馏(MOPD),一种用于组合多个领域RL教师能力的后训练范式:我们首先运行每个领域的专门RL以获得一组领域教师,然后将这些教师蒸馏到学生在其自身轨迹上的输出中。这消除了暴露偏差并提供了密集的优化信号。在Qwen3-30B-A3B上,MOPD优于Mix-RL、Cascade RL、Off-Policy Finetune和Param-Merge基线,几乎继承了每个教师的全部能力。MOPD还支持领域教师的并行独立开发,消除了多领域后训练中典型的跨领域耦合。MOPD已部署在工业级前沿模型MiMo-V2-Flash的后训练中,展示了其在前沿规模LLM中能力集成的实用价值。

英文摘要

Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.

URL PDF HTML 收藏