arXivDaily arXiv每日学术速递 周一至周五更新

大厂专区

Li Auto(理想汽车)

至 收录 120
2607.16339 2026-07-21 cs.AI 新提交

LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models

LaCache:用于扩散大语言模型的精确缓存和精度自适应推理

Xingru Chen, Zelang Liang, Yongjia Ma, Jiqing Zhan, Shuling Yang, Lian Wen, Kun Zhan

机构 * Li Auto Inc.(理想汽车公司)

AI总结 研究针对扩散大语言模型算子级冗余问题,提出LaCache加速框架,通过无损缓存中间结果及集成FP8量化策略,无需训练,单独使用可实现约1.3倍端到端加速,与现有方法结合可达40.2倍加速且保持任务精度。

详情
AI中文摘要

基于扩散的大语言模型(DLLMs)在文本生成中通过半自回归(SAR)解码实现并行生成。然而,当前方法存在严重的算子级冗余,在去噪步骤中会重新计算整个序列,忽略了前缀和掩码后缀在一个块内保持不变。我们提出了LaCache,一个无需训练的加速框架,通过无损缓存和混合精度来减轻这种冗余。具体来说,LaCache采用无损状态记忆(LSM),缓存三种类型的中间结果:用于嵌入输出的EmbedCache、用于逐令牌注意力前状态的RoPECache和用于FlashAttention内在线softmax统计的FACache。这些缓存允许模型在不变的令牌上跳过冗余计算而不改变输出。为进一步缓解内存带宽瓶颈,LaCache为FFN层集成了针对扩散过程中依赖步骤的激活分布定制的每组FP8量化策略。实验表明,单独使用LaCache比普通DLLM实现了约1.3倍的端到端加速。与现有加速方法结合时,LaCache在保持可比任务精度的同时达到了高达40.2倍的端到端加速。

英文摘要

Diffusion-based Large Language Models(DLLMs) enable parallel generation via Semi-Autoregressive (SAR) decoding in text generation. However, current methods suffer from severe operator-level redundancy: they recompute the entire sequence during denoising steps, ignoring that the prefix and masked suffix remain invariant within a block. We propose LaCache, a training-free acceleration framework that alleviates this redundancy through lossless caching and mixed precision. Specifically, LaCache employs Lossless State Memoization (LSM) by caching three types of intermediate results: (i) EmbedCache for embedding outputs, (ii) RoPECache for token-wise pre-attention states, and (iii) FACache for the online softmax statistics within FlashAttention. These caches allow the model to skip redundant computation on unchanged tokens without altering the output. To further alleviate memory-bandwidth bottlenecks, LaCache inegrates a per-group FP8 quantization strategy for FFN layers, tailored to step-dependent activation distributions across the diffusion process. Experiments demonstrate that LaCache alone achieves approximately 1.3X end-to-end speedup over vanilla DLLM. When combined with existing acceleration methods, LaCache reaches up to 40.2X end-to-end speedup while maintaining comparable task accuracy.

URL PDF HTML 收藏
2607.09375 2026-07-13 cs.LG cs.CL 新提交

Mach-Mind-4-Flash Technical Report

马赫思维-4-闪技术报告

Foundation Model Team

机构 * Li Auto Inc(理想汽车公司)

AI总结 马赫思维-4-闪是含3B激活参数的35B参数专家混合智能体模型,通过训练后优化及可扩展交互环境提升性能。其流程含三个阶段,在多个基准测试中成绩优异,以低推理成本领先或匹配数倍激活规模模型。

详情
AI中文摘要

我们提出了马赫思维-4-闪,这是一个具有3B激活参数的35B参数的专家混合(MoE)智能体模型。仅通过训练后优化而不扩大预训练计算规模,该模型就达到了与100B参数级模型相当或超越其性能。通过为大规模强化学习引入可扩展的智能体交互环境,该模型在实际应用任务中取得了显著的性能提升。我们的流程包括三个阶段:统一的强化学习/优化训练基础设施,多领域强化学习专家并行训练后融合,以及混合中位数长度策略优化。马赫思维-4-闪在多个基准测试中取得了优异成绩。

英文摘要

We present Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts (MoE) agentic model with 3B activated parameters. Through post-training optimization alone without scaling pre-training compute, the model achieves performance on par with or surpassing that of 100B-parameter-class models. By introducing scalable agentic interaction environments for large-scale reinforcement learning, the model attains significant performance gains on real-world application tasks. Our pipeline comprises three stages: (1) a unified RL/OPD training infrastructure with dynamic multi-teacher scheduling and operator-level acceleration, delivering 17\% end-to-end training speedup; (2) multiple domain-specific RL experts trained in parallel across Reasoning, General, and Agent tracks, then fused into a single generalist via Multi-Teacher On-Policy Distillation (MOPD) -- a routed reverse-KL objective that eliminates the see-saw degradation of mixed-reward RL; (3) Hybrid Median-length Policy Optimization (HMPO), a single-stage token-efficiency method that compresses reasoning chains by 19--46\% with $\le$0.7 percentage-point accuracy loss. Mach-Mind-4-Flash scores 92.70 on AIME'26, 82.82 on IFBench, 80.74 on Behavioral-SafetyBench, 75.80 on BFCL-v4, 72.31 on BrowseComp-zh, and 84.20 on ClawBench -- leading or matching models with 10--30$\times$ its activated size at a fraction of the inference cost.

URL PDF HTML 收藏
2601.22054 2026-07-07 cs.CV cs.AI 版本更新

MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources

MetricAnything:利用噪声异构源扩展度量深度预训练

Baorui Ma, Jiahui Yang, Donglin Di, Xuancheng Zhang, Jianxun Cui, Hao Li, Yan Xie, Wei Chen

机构 * Li Auto Inc(李自动公司)

AI总结 研究针对度量深度估计扩展难题,提出MetricAnything框架,通过随机掩码创建稀疏度量提示解耦空间推理与偏差,用多样3D数据预训练,在多任务中表现出色,推动度量感知发展。

Comments Accepted to ECCV 2026. Project Page: https://metric-anything.github.io/metric-anything-io/

详情
AI中文摘要

扩展助力视觉基础模型进步,但将此范式应用于度量深度估计具挑战性。我们引入MetricAnything,一个简单可扩展的预训练框架,无需人工提示等学习度量深度。核心是稀疏度量提示,用约20M图像-深度对展示了度量深度的缩放趋势,预训练模型在多任务中表现优异,开源框架支持研究。

英文摘要

Scaling has powered recent advances in vision foundation models, yet extending this paradigm to metric depth estimation remains challenging due to heterogeneous sensor noise, camera-dependent biases, and metric ambiguity in noisy cross-source 3D data. We introduce Metric Anything, a simple and scalable pretraining framework that learns metric depth from noisy, diverse 3D sources without manually engineered prompts, camera-specific modeling, or task-specific architectures. Central to our approach is the Sparse Metric Prompt, created by randomly masking depth maps, which serves as a universal interface that decouples spatial reasoning from sensor and camera biases. Using about 20M image-depth pairs spanning reconstructed, captured, and rendered 3D data across 10000 camera models, we demonstrate-for the first time-a clear scaling trend in the metric depth track. The pretrained model excels at prompt-driven tasks such as depth completion, super-resolution and Radar-camera fusion, while its distilled prompt-free student achieves state-of-the-art results on monocular depth estimation, camera intrinsics recovery, single/multi-view metric 3D reconstruction, and VLA planning. We also show that using pretrained ViT of Metric Anything as a visual encoder significantly boosts Multimodal Large Language Model capabilities in spatial intelligence. These results show that metric depth estimation can benefit from the same scaling laws that drive modern foundation models, establishing a new path toward scalable and efficient real-world metric perception. We open-source MetricAnything at http://metric-anything.github.io/metric-anything-io/ to support community research.

URL PDF HTML 收藏
2607.01586 2026-07-03 cs.CV cs.AI cs.RO 新提交

VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment

VLAFlow:通过协同训练和未来潜在对齐的视觉-语言-动作模型统一训练框架

Guoyang Xia, Fengfa Li, Hongjin Ji, Lei Ren, Fangxiang Feng, Kun Zhan, Yan Xie

机构 * Li Auto Inc.(理想汽车) School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院) The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

AI总结 提出VLAFlow统一框架,比较四种训练范式,发现语言监督保持视觉-语言泛化,未来潜在对齐改进状态转换建模,两者结合实现最佳迁移性能。

详情
AI中文摘要

视觉-语言-动作模型(VLA)最近推动了机器人操作的发展,然而不同机器人数据预训练范式的影响仍然难以比较,因为现有模型在架构、数据、动作空间和评估协议上往往不同。我们提出VLAFlow(视觉-语言-动作流),一个用于VLA训练目标受控比较的统一流匹配框架。使用包含来自DROID、OpenX-Embodiment、OpenX-Augmented和RoboCOIN的约5000小时数据的异构机器人语料库OXEMix,我们在相同的pi0风格架构、共享VLM骨干、动作专家和14维动作空间下评估了四种范式:纯动作建模(MindPI)、语言监督协同训练(MindLPI)、未来潜在对齐(MindWPI)及其组合(MindLWPI)。在LIBERO、LIBERO-Plus和SimplerEnv上的实验表明,纯动作预训练对异构数据敏感。相比之下,语言监督有助于保持视觉-语言泛化,而未来潜在对齐改进了状态转换和动作结果建模。通过结合这两种信号,MindLWPI在基准测试中实现了最稳定的整体迁移性能。这些结果表明了一种元动作空间观点:语言和未来潜在表示提供了互补的中间约束,使得异构动作监督更加平滑和可迁移。

英文摘要

Vision-language-action models (VLAs) have recently advanced robotic manipulation, yet the effects of different robot-data pre-training paradigms remain difficult to compare because existing models often differ in architecture, data, action space, and evaluation protocol. We present VLAFlow (Vision-Language-Action Flow), a unified flow-matching framework for controlled comparison of VLA training objectives. Using a heterogeneous robot corpus, OXEMix, containing approximately 5,000 hours of data from DROID, OpenX-Embodiment, OpenX-Augmented, and RoboCOIN, we evaluate four paradigms under the same pi0-style architecture, shared VLM backbone, action expert, and 14-dimensional action space: action-only modeling (MindPI), language-supervised co-training (MindLPI), future latent alignment (MindWPI), and their combination (MindLWPI). Experiments on LIBERO, LIBERO-Plus, and SimplerEnv show that action-only pre-training is sensitive to heterogeneous data. In contrast, language supervision helps preserve vision-language generalization, while future latent alignment improves state-transition and action-outcome modeling. By combining both signals, MindLWPI achieves the most stable overall transfer performance across benchmarks. These results suggest a meta-action space view: language and future latent representations provide complementary intermediate constraints that make heterogeneous action supervision smoother and more transferable.

URL PDF HTML 收藏
2606.20083 2026-07-02 cs.CV 新提交

Holo-World: Unified Camera, Object and Weather Control for Video World Model

Holo-World: 视频世界模型的统一相机、物体和天气控制

Xiangchen Yin, Wenzhang Sun, Jiahui Yuan, Zijie Liu, Yinda Chen, Wei Li, Dachun Kai, Chunfeng Wang, Xiaoyan Sun

机构 * University of Science and Technology of China(中国科学技术大学) Li Auto Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合国家科学中心人工智能研究院)

AI总结 提出Holo-World,一种从单张图像联合控制相机、物体运动和天气的统一视频世界模型,通过场景适配器和解耦CFG实现世界保持与天气迁移。

Comments Project Page: https://xiangchenyin.github.io/Holo-World Code: https://github.com/XiangchenYin/Holo-World

详情
AI中文摘要

视频世界模型正朝着在可控相机和物体运动下保持观察到的世界,同时允许其环境状态变化的方向发展。然而,这些控制仍然是孤立的,天气生成通常依赖于已经指定未来结构的源视频或重建场景。我们研究了一种基于第一帧锚定的源到状态设置,其中模型从单张图像开始,遵循明确的相机和物体控制以及可选的天气指令,然后生成一个视频,该视频要么保持源世界,要么将其转移到目标天气状态。为了解决这些挑战,我们首先构建了HoloStateData,一个状态视频数据集,将多样化的视频转换为用于相机、物体和天气监督的统一控制样本。其次,我们引入了Holo-World,一个统一的、可控制的视频世界模型,从单张图像联合控制场景。其统一场景适配器将世界保持和天气迁移分解为不同的参数子空间,使用渲染背景、几何缓冲区和物体控制来维持受控场景结构,同时建模依赖天气的外观和粒子效果。此外,场景-天气解耦CFG分别引导场景和天气残差,增强目标天气效果而不过度放大完整条件。定量和定性实验表明,Holo-World在保持精确的相机和物体控制以及一致场景结构的同时,将场景迁移到多样化的目标天气状态,在天气状态生成上优于视频到视频的天气编辑基线。我们的项目页面可在\url{this https URL}获取。

英文摘要

Video world models are moving toward preserving an observed world under controllable camera and object motion while allowing its environmental state to change. Yet these controls remain isolated, and weather generation typically relies on a source video or reconstructed scene that already specifies future structure. We study a first-frame-anchored source-to-state setting, where the model starts from a single image and follows explicit camera and object controls and an optional weather instruction, then generates a video that either preserves the source world or transfers it to a target weather state. To address these challenges, we first build HoloStateData, a state video dataset that turns diverse videos into unified control samples for camera, object, and weather supervision. Second, we introduce Holo-World, a unified controllable video world model that jointly controls the scene from a single image. Its Unified Scene Adapter factorizes world preservation and weather transfer into distinct parameter subspaces, using rendered background, geometry buffers, and object controls to maintain controlled scene structure while modeling weather-dependent appearance and particle effects. Additionally, Scene-Weather Decomposed CFG guides scene and weather residuals separately, strengthening target weather effects without over-amplifying the full condition. Quantitative and qualitative experiments demonstrate that Holo-World maintains precise camera and object controls with consistent scene structure while transferring scenes into diverse target weather states, outperforming video-to-video weather editing baselines on weather-state generation. Our project page is available at https://xiangchenyin.github.io/Holo-World/

URL PDF HTML 收藏
2606.31383 2026-07-01 cs.CV 新提交

MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs

MS-Resampler: 用于高效多模态大语言模型的多范围视觉重采样

Zhongyang Li, Yaqian Li, Faming Fang, Rinyoichi Takezoe, Zi-Hao Bo, Cheng Qian, Mo Guang, Guixu Zhang, Kaiwen Long

机构 * Li Auto Inc.(理想汽车) East China Normal University(华东师范大学)

AI总结 提出MS-Resampler,通过多范围视觉重采样框架,注入显式空间范围先验,使模型在固定令牌预算内同时捕获局部细节和全局上下文,提升多模态理解与推理。

详情
AI中文摘要

多模态大语言模型(MLLMs)通常采用基于重采样的投影器,将密集的视觉特征转换为紧凑的令牌序列以进行语言建模。大多数现有的重采样器通过全局交叉注意力采用单一的固定聚合范围,这可能会模糊细粒度的局部证据,并限制在固定令牌预算内捕获局部细节和全局上下文的能力。在这项工作中,我们提出了MS-Resampler,一种用于MLLMs的多范围视觉重采样框架。MS-Resampler通过将显式的空间范围先验注入重采样注意力,实例化了多个特定范围的采样器,使每个分支能够从局部到全局的特定粒度聚合视觉信息。然后,这些特定范围采样器的输出被自适应地融合,以生成用于语言建模的最终视觉表示。在十个公共多模态基准上的大量实验表明,MS-Resampler在仅引入最小计算开销的情况下,一致地提升了视觉理解和多模态推理能力,优于传统的单范围重采样器。

英文摘要

Multimodal large language models (MLLMs) typically employ resampling-based projectors to transform dense visual features into a compact token sequence for language modeling. Most existing resamplers adopt a single, fixed aggregation scope via global cross-attention, which can blur fine-grained local evidence and limit the ability to capture both local details and global context within a fixed token budget. In this work, we propose MS-Resampler, a multi-scope visual resampling framework for MLLMs. MS-Resampler instantiates multiple scope-specific resamplers by injecting explicit spatial scope priors into the resampling attention, enabling each branch to aggregate visual information at a particular granularity from local to global. The outputs of these scope-specific resamplers are then adaptively fused to produce the final visual representations for language modeling. Extensive experiments on ten public multimodal benchmarks show that MS-Resampler consistently improves visual understanding and multimodal reasoning over conventional single-scope resamplers, while introducing only minimal computational overhead.

URL PDF HTML 收藏
2606.29869 2026-06-30 cs.CL cs.AI

ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation

ARKD: 自适应强化学习引导的双向KL散度蒸馏用于文本生成

Zilong Liu, Xuewen Zhang, Jinrui Xing, Juyi Qiao, Huiyong Wang, Junming Jiao

机构 * Li Auto, Beijing, China(北京利亚自动化有限公司) School of Software and Microelectronics, Peking University, China(北京大学软件与微电子学院)

AI总结 针对单一KL目标难以平衡主分布拟合与长尾概率建模的问题,提出基于强化学习的自适应KL加权蒸馏框架,通过策略网络动态分配前向和反向KL散度权重,实现主模态和长尾模态的双重对齐,在Rouge-L和BertScore上持续提升。

详情
AI中文摘要

知识蒸馏(KD)是压缩大型语言模型(LLM)的关键技术,然而依赖单一KL目标的方法往往难以平衡主分布拟合与长尾概率建模,限制了生成质量和泛化能力。为解决这一问题,我们从理论和实证角度分析了前向和反向KL散度(FKL/RKL)在分布对齐中的互补作用。随后,我们提出了一种基于强化学习的自适应KL加权蒸馏框架,其中策略网络根据师生分布特征动态分配FKL和RKL的权重,并由即时奖励信号引导,实现主模态和长尾模态的双重对齐。大量实验表明,该方法在Rouge-L和BertScore指标上持续改进,比贪婪启发式方法高出0.4-0.6个点,并在多个基准上优于其他基线方法。

英文摘要

Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization. To address this, we analyze the complementary roles of forward and reverse KL divergence (FKL/RKL) in distribution alignment from theoretical and empirical perspectives. We then propose a reinforcement-learning-based adaptive KL-weighted distillation framework, in which a policy network dynamically assigns weights to FKL and RKL based on teacher-student distributional characteristics, guided by immediate reward signals to achieve dual alignment on principal and long-tail modes. Extensive experiments demonstrate consistent improvements across Rouge-L and BertScore metrics, surpassing greedy heuristics by 0.4-0.6 points and outperforming other baseline methods on diverse benchmarks.

URL PDF HTML 收藏
2603.09731 2026-06-29 cs.CV cs.AI cs.CL 版本更新

EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning

EXPLORE-Bench:具有长视距推理的自我中心场景预测

Chengjun Yu, Xuhan Zhu, Chaoqun Du, Pengfei Yu, Wei Zhai, Yang Cao, Zheng-Jun Zha

机构 * University of Science and Technology of China(中国科学技术大学) Foundation Model Team, Li Auto Inc.(蔚来汽车基础模型团队) Tsinghua University(清华大学)

AI总结 提出EXPLORE-Bench基准,通过初始场景图像和动作序列预测最终场景,评估多模态大模型在自我中心长视距推理中的能力,发现与人类存在显著差距。

详情
AI中文摘要

多模态大语言模型(MLLMs)越来越被视为具身智能体的基础,但目前尚不清楚它们能否从自我中心视角可靠地推理动作的长期物理后果。我们通过一个新任务——具有长视距推理的自我中心场景预测来研究这一差距:给定初始场景图像和一系列原子动作描述,模型被要求预测所有动作执行后的最终场景。为了实现系统评估,我们引入了EXPLORE-Bench,这是一个从涵盖多种场景的真实第一人称视频中整理的基准。每个实例将长动作序列与结构化的最终场景注释配对,包括对象类别、视觉属性和对象间关系,支持细粒度的定量评估。在一系列专有和开源MLLMs上的实验揭示了与人类之间的显著性能差距,表明长视距自我中心推理仍然是一个重大挑战。我们进一步通过逐步推理分析了测试时扩展,并表明分解长动作序列可以在一定程度上提高性能,同时带来非平凡的计算开销。总体而言,EXPLORE-Bench为测量和推进自我中心具身感知的长视距推理提供了一个原则性测试平台。

英文摘要

Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint. We study this gap through a new task, Egocentric Scene Prediction with LOng-horizon REasoning: given an initial-scene image and a sequence of atomic action descriptions, a model is asked to predict the final scene after all actions are executed. To enable systematic evaluation, we introduce EXPLORE-Bench, a benchmark curated from real first-person videos spanning diverse scenarios. Each instance pairs long action sequences with structured final-scene annotations, including object categories, visual attributes, and inter-object relations, which supports fine-grained, quantitative assessment. Experiments on a range of proprietary and open-source MLLMs reveal a significant performance gap to humans, indicating that long-horizon egocentric reasoning remains a major challenge. We further analyze test-time scaling via stepwise reasoning and show that decomposing long action sequences can improve performance to some extent, while incurring non-trivial computational overhead. Overall, EXPLORE-Bench provides a principled testbed for measuring and advancing long-horizon reasoning for egocentric embodied perception.

URL PDF HTML 收藏
2606.19186 2026-06-19 cs.RO cs.LG 新提交

Learning to Annotate Delayed and False AEB Events: A Practical System for Extreme Class Imbalance and Asymmetric Label Noise

学习标注延迟和误报AEB事件:针对极端类别不平衡和非对称标签噪声的实用系统

Mengxiang Hao, Xin Jiang, Xinghao Huang, Wenliang Su, Zhiteng Wang, Junjie Rao, Xiaotian Yang, Wei Liao, Chengyu Han, Gen Liang, Yulun Song, Zhitao Xu, Xianpeng Lang

机构 * Li Auto

AI总结 提出首个自动化AEB标注框架,通过特定数据增强和噪声抑制技术,解决极端类别不平衡和非对称标签噪声问题,将延迟/误报触发召回率提升80%,人工工作量减少50%。

Comments 8 pages, 5 figures, accepted by IEEE International Conference on Robotics and Automation (ICRA)

Journal ref 2026 IEEE International Conference on Robotics and Automation (ICRA)

详情
AI中文摘要

自主紧急制动(AEB)优化依赖于准确标注的真实世界触发事件,特别是揭示系统缺陷的罕见但关键的延迟和误报AEB触发事件。然而,这些少数样本在每天数千次触发事件中占比不到5%,使得大规模人工标注成本过高。我们提出了首个自动化AEB标注框架来解决这一问题。在开发过程中,我们识别出两个严重损害延迟/误报触发标注准确性的基本挑战:(1)极端类别不平衡,其中延迟/误报触发被真实触发淹没;(2)非对称标签噪声,其中误标注的多数样本(真实触发)抑制了少数样本(延迟/误报触发)的学习。为克服这些挑战,我们提出两项关键创新:(1)特定数据增强,通过操纵焦点目标属性、移植自车动态和掩蔽非焦点代理来合成逼真样本;(2)噪声抑制,使用稳定硬度估计和探针引导的自适应阈值来清理误标注的真实触发样本。关键的是,我们将模型部署为具有全栈架构的实用标注系统,从每天数千个AEB事件中高效识别关键的延迟/误报触发。生产结果表明,延迟/误报触发的召回率提高了80%,人工工作量减少了50%。除了直接收益,该系统通过积累高质量标注实现持续自我改进,为车载AEB系统优化奠定了必要的数据基础。

英文摘要

Autonomous Emergency Braking (AEB) optimization relies on accurately annotated real-world trigger events, particularly rare but critical delayed and false AEB triggers that expose system deficiencies. However, these minority samples comprise less than 5% of thousands of daily triggers, making manual annotation prohibitively expensive at scale. We present the first automated AEB annotation framework to address this problem. During development, we identified two fundamental challenges that severely impair delayed/false trigger annotation accuracy: (1) Extreme class imbalance where delayed/false triggers are overwhelmed by true triggers; (2) Asymmetric label noise where mislabeled majority samples (true triggers) suppress minority samples (delayed/false triggers) learning. To overcome these challenges, we propose two key innovations: (1) Specific data augmentation that synthesizes realistic samples by manipulating focal target attributes, transplanting ego-vehicle dynamics, and masking non-focal agents; (2) noise suppression using stable hardness estimation and probe-guided adaptive threshold to clean mislabeled true trigger samples. Crucially, we deploy our model as a practical annotation system with full-stack architecture, efficiently identifying critical delayed/false triggers from thousands of daily AEB events. Production results demonstrate 80% improvement in recall of delayed/false triggers and 50% reduction in manual workload. Beyond immediate gains, the system enables continuous self-improvement through accumulated high-quality annotations, establishing a necessary data foundation for on-vehicle AEB system optimization

URL PDF HTML 收藏
2606.18864 2026-06-18 cs.LG cs.AI 新提交

Scaling Learning-based AEB with Massive Unlabeled Data

基于大规模无标签数据的可扩展学习型自动紧急制动

Xiangyu Wang, Yang Zhan, Mengxiang Hao, Chuanchuan Zhong, Yansong Jia, Junjie Zhang, Yu Han, Xin Jiang, Zhen Cao, Ying Wang, Yulun Song, Zhitao Xu

机构 * Li Auto

AI总结 提出稳定元反馈半监督学习框架,通过噪声感知解耦和运动学门控伪标签,利用大规模无标签数据提升自动紧急制动性能,实现超100:1正误触发比和35%无事故里程提升。

Comments Accepted for presentation at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

详情
AI中文摘要

本文研究如何在生产约束下,利用大规模无标签车队数据扩展基于学习的自动紧急制动(AEB)。我们的方法基于元反馈半监督学习(MF-SSL),其中教师模型为无标签驾驶数据生成伪标签,并使用小型有标签锚定集作为安全关键反馈进行更新。在生产中,锚定歧义和有标签-无标签不匹配会放大系统性的伪标签错误,导致误触发。我们提出了一种稳定的MF-SSL框架,包括:(i) 噪声感知解耦,从教师监督更新路径中移除易产生歧义的锚定;(ii) 运动学门控伪标签,结合教师冲突惩罚,抑制无标签数据上由不匹配引起的风险幻觉,同时保持广泛覆盖。大量实验表明,随着无标签数据从1M扩展到1B窗口,模型性能持续提升,在保持舒适性的同时提高了安全性。经过1B数据训练的学生模型已部署到数十万辆车辆上,并在超过10^9公里的行驶中得到验证,实现了超过100:1的正误触发比,且相比仅基于规则的基线,无事故行驶里程提升了35%。

英文摘要

This paper studies how to scale learning-based automatic emergency braking (AEB) with massive unlabeled fleet data under production constraints. Our approach is based on meta-feedback semi-supervised learning (MF-SSL), where a teacher generates pseudo labels for unlabeled driving data and is updated using a small labeled anchor set as safety-critical feedback. In production, anchor ambiguity and labeled-unlabeled mismatch can amplify systematic pseudo-label errors, leading to spurious triggers. We propose a stabilized MF-SSL framework with (i) Noise-Aware Decoupling, which removes ambiguity-prone anchors from the teacher's supervised update path, and (ii) kinematics-gated pseudo-labeling with a teacher conflict penalty to suppress mismatch-induced risk hallucinations on unlabeled data while maintaining broad coverage. Extensive experiments show consistent gains as unlabeled data scale from 1M to 1B windows, improving safety while keeping comfort stable. The 1B-trained student model is deployed to hundreds of thousands of vehicles and validated over \$10^9$ km of driving, achieving a positive-to-false activation ratio exceeding 100:1 and a 35% improvement in accident-free driving mileage over a production rule-only baseline.

URL PDF HTML 收藏
2606.16808 2026-06-16 cs.AI 新提交

Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Models

自适应且显式安全:触发大型推理模型中的潜在安全意识

Ke Miao, Jiaxin Li, Hongliang Chen, Yuke Hu, Zhan Qin

机构 * The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室) Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China(杭州高新区(滨江)区块链与数据安全研究院) Li Auto Inc.(理想汽车) Tsinghua University(清华大学) King Abdullah University Of Science And Technology(阿卜杜拉国王科技大学)

AI总结 针对大型推理模型易受越狱攻击的问题,提出Safe Trigger方法,通过SFT显式诱导安全标签触发安全分析,并用DPO优化,显著降低攻击成功率而不影响通用性能。

详情
AI中文摘要

尽管大型推理模型(LRMs)在复杂任务上表现出色,但它们仍然极易受到复杂的越狱攻击和直接的有害查询。为了解决这一脆弱性,先前的工作严重依赖外部手动数据注释进行安全对齐。然而,我们观察到,当原始查询与其自身的推理轨迹一起重新呈现时,LRMs能够固有地识别安全风险——我们将这种能力称为潜在安全意识。为了利用这种安全意识,我们首先采用监督微调(SFT)显式诱导安全标签,以在初始推理内容之后触发对不安全查询的安全分析和指导,同时保留对一般查询的标准响应以确保自适应触发。随后,我们应用直接偏好优化(DPO)进一步增强安全分析和指导的正确性和稳定性。值得注意的是,两个训练阶段所需的响应完全由正在优化的模型生成。通过(Safe Trigger)SFT和DPO,实验结果表明安全性显著增强。例如,DeepSeek-R1-Distill-Llama-8B在有害和越狱基准上的平均攻击成功率(ASR)分别下降了24.65%和36.72%。最后,我们的Safe Trigger方法对通用性能或用户体验几乎没有负面影响。

英文摘要

While Large Reasoning Models (LRMs) excel at complex tasks, they remain highly vulnerable to sophisticated jailbreaks and direct harmful queries. To address this vulnerability, prior works depend heavily on external manual data annotation for safety alignment. However, we observe that LRMs can inherently identify safety risks when being re-presented with original queries alongside their own reasoning trajectories -- a capability we term Latent Safety Awareness. To leverage this safety awareness, we first employ Supervised Fine-Tuning (SFT) to explicitly induce safe tags to trigger safety analysis and guidance following the initial reasoning content for unsafe queries, while preserving standard responses for general queries to ensure adaptive triggering. Subsequently, we apply Direct Preference Optimization (DPO) to further enhance the correctness and stability of the safety analysis and guidance. Notably, responses required for both training stages are entirely generated by models being optimized. With (Safe Trigger) SFT and DPO, experimental results demonstrate significant safety enhancement. For example, the Attack Success Rate (ASR) of DeepSeek-R1-Distill-Llama-8B, on average, drops 24.65% and 36.72% on harmful and jailbreak benchmarks, respectively. Finally, our Safe Trigger method exerts almost no negative impact on general performance or user experience.

URL PDF HTML 收藏
2606.15869 2026-06-16 cs.CV 新提交

Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation

Metis: 一种用于自动驾驶和城市导航的通用高效世界-动作模型

Jingyu Li, Zhe Liu, Dongnan Hu, Junjie Wu, Zipei Ma, Wenxiao Wu, Chao Han, Zhihui Hao, Zhikang Liu, Kun Zhan, Jiankang Deng, Xiatian Zhu, Li Zhang

机构 * Fudan University(复旦大学) Shanghai Innovation Institute(上海创新研究院) The University of Hong Kong(香港大学) Tongji University(同济大学) Li Auto Inc.(理想汽车) Huazhong University of Science and Technology(华中科技大学) Imperial College London(伦敦帝国理工学院) University of Surrey(萨里大学)

AI总结 提出Metis框架,通过解耦视频生成与动作预测,采用混合专家架构和不对称注意力掩码,实现高效推理与泛化,在多个导航基准上取得最优性能。

详情
AI中文摘要

世界-动作模型(WAMs)在自动驾驶和城市导航中展现出巨大潜力。基于视觉-语言-动作模型或视频生成模型的现有方法存在关键限制:(1)测试时因预测未来观测而导致高推理延迟,(2)视频与动作建模紧密耦合导致表示不匹配和泛化能力下降。为解决这两个问题,我们提出Metis,一种端到端WAM框架,将视频生成与动作预测解耦。具体而言,Metis采用混合专家(Mixture-of-Transformers)架构,包含专门用于视频生成和动作预测的专家,保留了每个任务的内在分布特性。为提高效率,我们引入非对称注意力掩码,使得两个专家能够联合训练,同时允许动作模型在推理时绕过显式视频生成。这种设计确保了训练-推理一致性,并在不牺牲规划性能的情况下显著降低计算成本。大量实验表明,Metis在NAVSIM navhard和navtest基准以及CityWalker导航基准上取得了最先进的性能,验证了其在多样化任务中的泛化能力和效率。真实机器人部署进一步证实了我们方法的实际可行性。

英文摘要

World action models~(WAMs) have shown great promise for autonomous driving and urban navigation. Built upon Vision-Language-Action models or video generation models, existing approaches suffer key limitations: (1) High inference latency due to future observation prediction at test time, and (2) tightly coupled video and action modeling leading to representational mismatch and degraded generalization. To address both issues, we propose Metis, an end-to-end WAM framework that decouples video generation and action prediction. Specifically, Metis employs a Mixture-of-Transformers architecture with dedicated experts for video generation and action prediction, preserving the intrinsic distributional properties of each task. To enhance efficiency, we introduce an asymmetric attention mask that enables joint training of both experts while allowing the action model to bypass explicit video generation during inference. This design ensures training-inference consistency and significantly reduces computational costs without compromising planning performance. Extensive experiments demonstrate state-of-the-art performance on the NAVSIM navhard and navtest benchmarks and the CityWalker navigation benchmark, validating both the generalizability and efficiency across diverse tasks. Real-robot deployments further confirm the practical feasibility of our approach.

URL PDF HTML 收藏
2606.05774 2026-06-15 cs.CV 版本更新

LiAuto-GeoX: Efficient Grounded Driving Transformer

LiAuto-GeoX: 高效接地驾驶Transformer

Jiawei Lian, Haoyi Sun, Yang Wu, Lifu Mu, Siyuan Wang, Le Hui, Ning Mao, Tao Wei, Pan Zhou, Kun Zhan, Jian Yang

机构 * Nanjing University of Science and Technology(南京理工大学) Li Auto Inc.(Li Auto公司) Northwestern Polytechnical University(西北工业大学) Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算学院)

AI总结 提出LiAuto-GeoX,通过稀疏激光雷达先验和几何保持蒸馏框架,实现高效、实时的自车中心密集3D重建,并显著提升下游自动驾驶任务性能。

详情
AI中文摘要

密集3D重建在空间理解方面展现出巨大潜力,但其作为自动驾驶实时车载表示的可行性仍是一个开放挑战。现有大规模视觉几何模型通常需要大量计算资源,且缺乏动态驾驶环境所需的远距离几何保真度、环视一致性和实时效率。为弥补这一差距,我们提出 extbf{LiAuto-GeoX},一种为可部署的自车中心3D场景理解设计的高效接地驾驶Transformer。我们的方法首先从大规模环视数据中学习高容量驾驶几何模型,利用稀疏激光雷达先验在远处、模糊或结构稀疏区域提供稳健的几何接地。然后,通过一种新颖的几何保持蒸馏框架,将这一能力实例化为高度紧凑的1.55亿参数车载模型。该框架采用掩码引导的深度感知蒸馏,通过强调几何信息丰富的区域来保留细粒度度量结构,以及相对姿态关系蒸馏,通过姿态诱导的几何关系强制跨视图空间一致性。大量评估表明, extbf{LiAuto-GeoX}在KITTI上以220 FPS运行,同时保持高保真密集重建,实现实时部署。学习到的几何结构无缝迁移到下游自主任务,在轨迹预测中达到90.6 PDMS,在占用预测中达到24.63 mIoU,在未来帧预测中达到47.67 IoU。这些结果表明,高效的密集3D重建可以超越其作为感知目标的传统角色,作为下一代自动驾驶的可扩展基础几何表示。

英文摘要

Dense 3D reconstruction has demonstrated immense potential for spatial understanding, yet its viability as a real-time, onboard representation for autonomous driving remains an open challenge. Existing large-scale visual geometry models typically require substantial computational resources and lack the long-range geometric fidelity, surround-view consistency, and real-time efficiency demanded by dynamic driving environments. To bridge this gap, we present \textbf{LiAuto-GeoX}, an efficient grounded driving transformer designed for deployable, ego-centric 3D scene understanding. Our approach begins by learning a high-capacity driving geometry model from large-scale surround-view data, utilizing sparse LiDAR priors to provide robust geometric grounding in distant, ambiguous, or structure-sparse regions. We then instantiate this capability into a highly compact 155M-parameter onboard model through a novel geometry-preserving distillation framework. This framework employs mask-guided depth-aware distillation to retain fine-grained metric structures by emphasizing geometrically informative regions, and relative-pose relational distillation to enforce cross-view spatial consistency through pose-induced geometric relations. Extensive evaluations reveal that \textbf{LiAuto-GeoX} runs at 220 FPS on KITTI while maintaining high-fidelity dense reconstruction, enabling real-time deployment. The learned geometry transfers seamlessly to downstream autonomy tasks, achieving 90.6 PDMS in trajectory prediction, 24.63 mIoU in occupancy prediction, and 47.67 IoU in future-frame prediction. These all demonstrate that efficient dense 3D reconstruction can transcend its traditional role as a perception target to serve as a scalable, foundational geometric representation for next-generation autonomous driving.

URL PDF HTML 收藏
2606.04387 2026-06-04 cs.IR cs.AI

Rethinking Sales Lead Scoring with LLM-based Hierarchical Preference Ranking

重新思考基于LLM的分层偏好排名的销售线索评分

Chenyu Zhang, Yiwen Liu, Yin Sun, Xinyuan Zhang, Yuji Cao, Junming Jiao, Juyi Qiao

机构 * Intelligent Business Team, Li Auto Inc.(李自动公司智能商务团队)

AI总结 针对高价值领域销售线索转化问题,提出基于LLM的判别式框架HPRO,通过分层偏好排名优化联合建模结构化与非结构化数据,实现评分与排名性能提升。

详情
AI中文摘要

在高价值领域(如汽车、房地产)中,销售线索转化与电子商务推荐有根本不同,因为其决策周期长且涉及多阶段漏斗。传统的线索评分方法(基于规则的评分卡、机器学习或逐点CTR模型)面临严重挑战:监督信号稀疏、非结构化CRM日志中的语义鸿沟,以及无法捕捉线索的相对优先级。虽然大型语言模型(LLM)能够对客户交互提供卓越的语义理解,但通用LLM不适合线索排名:它们生成文本而非可比较的分数,并且缺乏与销售漏斗分层优先级的一致性。我们提出了一种基于LLM的判别式框架用于销售线索评分,该框架支持结构化CRM特征和非结构化客户交互的联合建模。在此框架之上,我们提出了HPRO(分层偏好排名优化),通过分层偏好排名目标增强销售线索评分。HPRO采用边际感知的Bradley-Terry公式,将稀疏的二元标签转换为密集的、漏斗感知的偏好对,使线索评分能够同时利用逐点和成对监督。在来自领先新能源汽车品牌的大规模数据上的实验表明,分类性能达到最先进水平(AUC 0.8161),排名性能提升(排名靠前线索的精确度提高39.7%)。为期132天的在线A/B测试验证了9.5%的销量提升,确认了实际的商业影响。

英文摘要

Sales lead conversion in high-stakes domains (e.g., automotive, real estate) differs fundamentally from e-commerce recommendation due to prolonged decision cycles and multi-stage funnels. Traditional lead scoring methods rule-based scorecards, machine learning, or pointwise CTR models face severe challenges: sparse supervision, a semantic gap in unstructured CRM logs, and inability to capture relative lead priority. While Large Language Models(LLMs) offer superior semantic understanding of customer interactions, general-purpose LLMs are ill-suited for lead ranking: they generate text rather than comparable scores, and lack alignment with the hierarchical priorities of sales funnels. We introduce an LLM-based discriminative framework for sales lead scoring, which supports joint modeling of structured CRM features and unstructured customer interactions. On top of this framework, we propose HPRO (Hierarchical Preference Ranking Optimization), which augments sales lead scoring with a hierarchical preference ranking objective. HPRO employs a margin-aware Bradley-Terry formulation to transform sparse binary labels into dense, funnel-aware preference pairs, enabling lead scoring to leverage both pointwise and pairwise supervision. Experiments on large-scale data from a leading NEV brand demonstrate state-of-the-art classification (AUC 0.8161) and ranking performance (+39.7% precision among top-ranked leads). A 132-day online A/B test validates 9.5% sales volume uplift, confirming real-world commercial impact.

URL PDF HTML 收藏
2606.03318 2026-06-04 cs.CL

Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions

超越理想指令:现实交互中评估大语言模型的综合框架

Xuan Yang, Hao Xu, Tingfeng Hui, Hongsheng Xin, Kaike Zhang, Chunxiao Liu, Ning Miao

机构 * Department of Data Science, City University of Hong Kong(香港城市大学数据科学系) Hong Kong Institute of AI for Science, City University of Hong Kong(香港城市大学人工智能科学研究院) Li Auto Inc.(Li汽车公司) Beijing University of Posts and Telecommunications(北京邮电大学) Independent Researcher(独立研究员)

AI总结 提出RUT-Bench基准,通过高保真模拟理想与非理想用户行为,评估大语言模型在现实工具调用场景中的表现,发现所有测试模型成功率低于40%且面对复杂非理想输入时性能显著下降。

详情
AI中文摘要

尽管大语言模型(LLMs)在工具使用能力上取得了巨大进步,但现有的评估基准难以完全对齐真实世界场景。这些基准大多依赖于模拟的理想化用户假设,缺乏面向经验的评估。这些局限性未能考虑到真实用户特有的模糊性、不合作行为和意图转变。为填补这一空白,我们提出了RUT-Bench,一个专门用于评估LLMs在多样化真实用户工具调用场景下的基准。RUT-Bench支持高保真模拟,涵盖单轮和多轮对话中的理想理性模式和非理想异质行为。我们使用该基准对19个广泛采用的开源和专有LLMs进行了全面评估。实验结果显示,没有测试的LLM实现超过40%的整体成功率,并且几乎所有模型在面对更复杂的非理想用户输入时都经历了明显的性能下降。我们的代码和数据可在该https URL获取。

英文摘要

Despite great advances in tool-use capabilities of large language models (LLMs), existing evaluation benchmarks struggle to fully align with real-world scenarios. Such benchmarks mostly rely on simulated idealized user assumptions and lacks experience-oriented evaluation. These limitations fail to account for the ambiguity, uncooperative behaviors, and shifting intentions characteristic of real-world users. To fill this gap, we propose RUT-Bench, a dedicated benchmark designed to assess LLMs under diverse Real-world User Tool calling scenarios. RUT-Bench supports high-fidelity simulations covering both ideal rational patterns and heterogeneous non-ideal behaviors across single-turn and multi-turn dialogues. We conduct comprehensive evaluations on 19 widely adopted open-source and proprietary LLMs using our benchmark. Experimental results reveal that no tested LLMs achieve an overall success rate above 40%, and nearly all of them experience noticeable performance drops when facing more complicated non-ideal user inputs. Our code and data is available at https://github.com/Miaow-Lab/RUT-Bench.

URL PDF HTML 收藏
2606.01934 2026-06-02 cs.LG cs.CL

HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression

HMPO: 用于思维链压缩的混合中位数长度策略优化

Minghui Zheng, Hongxu Chen, Huimin Ren, Hongsheng Xin, Xiaoyang Qu, Ze Wang, Shuling Yang, Ziyu Peng, Kaike Zhang, Pan Zhou, Kun Zhan

机构 * Li Auto Inc.(Li Auto公司)

AI总结 提出HMPO,一种单阶段强化学习框架,通过自适应中位数预算、余弦衰减令牌奖励和乘法奖励公式,在数学数据上训练后实现19%-46%的令牌压缩且精度损失极小,并泛化至多种任务。

详情
AI中文摘要

大型语言模型通过扩展的思维链推理取得了显著性能,但这一冗长过程带来了大量推理开销。现有的思维链压缩方法面临不灵活的手动长度预算、计算昂贵的多阶段训练流程以及仅适用于小模型的脆弱可扩展性。我们提出HMPO(混合中位数长度策略优化),一种经济高效的单阶段强化学习框架。HMPO通过三个协同组件高效压缩思维链:基于成功轨迹的自适应中位数预算以消除手动调整、用于平滑长度惩罚的余弦衰减令牌奖励,以及通过严格优先考虑答案正确性来大幅减轻琐碎奖励破解的乘法奖励公式。仅在数学数据上训练,HMPO无缝泛化到数学、代码、科学和指令遵循任务。在从9B到122B参数、涵盖密集和混合专家架构的大规模实验中,HMPO实现了19%-46%的令牌压缩,精度下降可忽略,同时与现有的多阶段基线相比大幅降低了训练成本。

英文摘要

Large language models achieve remarkable performance via extended chain-of-thought (CoT) reasoning, yet this lengthy process incurs substantial inference overhead. Existing CoT compression methods struggle with inflexible manual length budgets, computationally expensive multi-stage training pipelines, and fragile scalability restricted to small models. We propose HMPO (Hybrid Median-length Policy Optimization), a cost-effective, single-stage reinforcement learning framework. HMPO efficiently compresses CoT via three synergistic components: an adaptive median-based budget derived from successful rollouts to eliminate manual tuning, a cosine-decay token reward for smooth length penalization, and a multiplicative reward formulation that substantially mitigates trivial reward hacking by strictly prioritizing answer correctness. Trained exclusively on mathematical data, HMPO generalizes seamlessly across math, code, science, and instruction-following tasks. Extensive experiments scaling from 9B to 122B parameters across dense and Mixture-of-Experts (MoE) architectures demonstrate that HMPO achieves 19%--46% token compression with negligible accuracy degradation, all while drastically reducing training costs compared to existing multi-stage baselines.

URL PDF HTML 收藏
2605.29697 2026-05-29 cs.AI

Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling

超越轨迹奖励:通过图建模实现智能搜索的步骤级信用分配

Yuchen Liu, Yingjie Feng, Lixiong Qin, Jiasi Chen, Jianing Yu, Sheng Gao, Sheng Yang, Weiran Xu

机构 * Beijing University of Posts and Telecommunications(北京邮电大学) Li Auto Inc.(李自动公司)

AI总结 针对智能搜索中轨迹级奖励无法量化单步行为贡献的问题,提出基于图距离贡献奖励(GDCR)的步骤级过程奖励,并结合步骤优势策略优化(SAPO)在四个基准上验证有效性。

Comments 15 pages, 8 figures

详情
AI中文摘要

在智能搜索中,轨迹级结果奖励无法量化单个步骤的行为贡献,而现有的步骤级奖励方法通常依赖于代价高昂的树采样。我们将世界知识视为潜在的世界图,并将每个信息搜索任务视为在潜在任务图中的搜索,其中有效步骤应朝着答案节点进行图进展。基于这一先验,我们提出图距离贡献奖励(GDCR),这是一种步骤级过程奖励,通过训练时实体-关系(ER)图中实体到答案节点的距离对新检索和引用的实体进行评分。我们进一步提出步骤优势策略优化(SAPO),它将GDCR转换为步骤级优势,并与轨迹级结果优势相结合。在四个具有挑战性的基准上的实验验证了我们方法的有效性。

英文摘要

In Agentic Search, trajectory-level outcome rewards fail to quantify the behavioral contributions of individual steps, while existing step-level reward methods typically rely on costly tree sampling. We view world knowledge as a latent world graph and each IS task as search within a latent task graph, where effective steps should make graph progress toward the answer node. Based on this prior, we propose Graph-Distance Contribution Reward (GDCR), a step-level process reward that scores newly-retrieved and newly-cited entities by their distance to the answer node in a training-time Entity-Relation (ER) graph. We further propose Step Advantage Policy Optimization (SAPO), which converts GDCR into step-level advantages and combines them with trajectory-level outcome advantages. Experiments on four challenging benchmarks validate the effectiveness of our method.

URL PDF HTML 收藏
2605.26496 2026-05-27 cs.LG cs.AI

Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling

Dense2MoE:通过统一剪枝和升级推动设备端LLM的帕累托前沿

Fengfa Li, Hongjin Ji, Yifeng Ding, Lei Ren, Chen Wei

机构 * Li Auto The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

AI总结 提出Dense2MoE框架,通过层融合升级(LF-UC)统一剪枝和升级,将密集LLM高效转换为设备端MoE模型,在推理延迟与准确性之间取得更优帕累托前沿。

Comments 19 pages

详情
AI中文摘要

混合专家(MoE)架构对于资源受限的设备端部署极具前景,但从头训练这些模型成本高昂。当前方法试图通过将密集模型升级为MoE来缓解这一问题,然而它们常常引入参数冗余,降低推理效率。另一方面,标准层剪枝减少了冗余,但不可避免地损害模型准确性。为解决这一困境,我们提出Dense2MoE,一种通过层融合升级(LF-UC)统一剪枝和升级的新框架。在硬件Roofline理论指导下,Dense2MoE通过剪枝来自冗余层的带宽密集型注意力模块,同时将其多层感知机(MLP)重新用作MoE专家,系统地克服了推理内存墙。这种结构创新保留了模型的核心能力,并通过选择性令牌路由严格限制活跃参数。借助适度的持续预训练预算,Dense2MoE高效地将公开可用的密集LLM转换为设备端就绪的MoE模型。大量实验表明,Dense2MoE显著推进了设备端推理延迟与模型准确性的帕累托前沿,优于密集基线、最先进的压缩方法和标准升级方法。

英文摘要

The Mixture of Experts MoE architecture is highly promising for resource constrained on device deployments yet training these models from scratch incurs prohibitive costs Current methods attempt to alleviate this by upcycling dense models into MoEs however they often introduce parameter redundancy that degrades inference efficiency Alternatively standard layer pruning mitigates redundancy but inevitably compromises model accuracy To resolve this dilemma we propose Dense2MoE a novel framework that unifies pruning and upcycling through Layer Fusion UpCycling LF UC Guided by hardware Roofline theory Dense2MoE systematically overcomes the inference memory wall by pruning bandwidth heavy attention modules from redundant layers while repurposing their Multi Layer Perceptrons MLPs into MoE experts This structural innovation preserves the models core capabilities and strictly limits active parameters via selective token routing With a modest continual pre training budget Dense2MoE efficiently converts publicly available dense LLMs into on device ready MoE models Extensive experiments demonstrate that Dense2MoE significantly advances the Pareto frontier for on device inference latency versus model accuracy outperforming dense baselines state of the art compression and standard upcycling methods

URL PDF HTML 收藏
2605.26113 2026-05-26 cs.RO cs.CV

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

AnyScene: 迈向高度可控的任意位置驾驶场景生成及超越

Haiming Zhang, Junfei Zhou, Feng Jiang, Jingzhong Li, Zhenglong Guo, Penglin Dai, Jifeng Dai, Yan Xie, Benjin Zhu

机构 * Li Auto(利汽车) Southwest Jiaotong University(西南交通大学) Tsinghua University(清华大学)

AI总结 提出AnyScene框架,通过时空占用扩散Transformer和几何引导视图扩展模块,实现从BEV布局生成语义占用序列和参考无关的多视角驾驶视频,支持精确可控和长时生成。

Comments Work in progress. Project page: https://mind-omni.github.io/

详情
AI中文摘要

生成高保真且可控的合成数据对于推进端到端自动驾驶至关重要,特别是解决罕见安全关键场景的长尾问题。现有的占用引导方法通常依赖于浅层条件机制和参考帧相关的视频合成,这限制了从任意BEV布局进行细粒度可控性,并限制了其在可扩展模拟中的适用性。在本文中,我们提出了AnyScene,一个统一的以占用为中心的驾驶场景生成框架。AnyScene通过时空占用扩散Transformer从BEV布局生成语义占用序列,该Transformer以自回归方式联合标记BEV和占用特征。这种设计使得从跨数据集和用户定义的BEV输入实现精确可控性,同时自然支持长时生成。基于生成的占用,几何引导视图扩展模块将占用视为规范空间表示,并以无参考和自回归方式合成时间一致的多视角驾驶视频,支持推理时的灵活相机配置。大量实验表明,AnyScene在占用和视频生成方面均达到最先进性能。它展现出对未见和定制布局的强大泛化能力,并为下游任务(如稀疏视图3D重建)提供可衡量的益处。

英文摘要

Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.

URL PDF HTML 收藏
2605.25334 2026-05-26 cs.CV

Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence

双路径几何感知多模态大语言模型用于空间智能

Yufei Zheng, Xuhan Zhu, Zide Liu, Chunpeng Zhou, Chenfeng Wang, Yongchao Xu, Yunnan Wang, Jiawei Liu, Pengfei Yu, Wei Zhai, Yang Cao, Zheng-Jun Zha

机构 * University of Science and Technology of China(中国科学技术大学) Li Auto Inc.(利汽车公司) Shanghai Jiao Tong University(上海交通大学)

AI总结 提出GAMSI,一种仅以RGB图像为输入、通过双路径查询和专家引导视觉对齐实现3D结构与度量尺度联合感知的多模态大语言模型,在七个空间智能基准上达到最优性能。

详情
AI中文摘要

从2D视觉输入理解物理世界的空间能力依赖于两种互补的几何知识:整体3D结构感知和细粒度度量尺度估计。现有的多模态大语言模型通常只处理其中一个方面,将深度图或点云作为额外模型输入,这带来了大量计算开销并继承了上游预测模型的泛化局限性。我们提出GAMSI,一种双路径几何感知多模态大语言模型用于空间智能,仅以RGB图像为输入,同时在统一的自回归骨干网络内内化两种几何先验。具体地,我们引入度量-结构解耦查询,使用两组可学习查询分别从共享视觉上下文中提取密集度量信号和稀疏结构线索,并通过任务解耦注意力掩码防止两条路径相互污染。在此基础上,专家引导视觉定位模块将聚合的线索投影回帧级视觉特征,并与视觉基础模型对齐,这些模型仅作为训练时的监督,而非模型输入。我们进一步构建了一个多任务空间指令微调数据集,包含152,776个样本,涵盖13种任务类型和三种视觉模态,整合自六个公共数据集。通过两阶段课程训练,GAMSI在七个空间智能基准上达到了最先进的性能。

英文摘要

Spatial understanding of the physical world from 2D visual inputs hinges on two complementary forms of geometric knowledge: holistic 3D structural perception and fine-grained metric scale estimation. Existing multimodal large language models (MLLMs) typically address only one facet, ingesting either depth maps or point clouds as additional model inputs, which incurs substantial computational overhead and inherits the generalization limitations of upstream prediction models. We propose GAMSI, a dual-pathway Geometry-Aware MLLM for Spatial Intelligence that takes only RGB images as input while internalizing both forms of geometric prior within a unified autoregressive backbone. Specifically, we introduce Metric-Structure Decoupled Queries (MSDQ) which employ two groups of learnable queries to respectively extract dense metric signals and sparse structural cues from the shared visual context, with a task-decoupled attention mask further preventing the two pathways from contaminating each other. Building on this, an Expert-Guided Visual Grounding (EVG) module projects the aggregated cues back to frame-level visual features and aligns them with vision foundation models, which serve purely as training-time supervision, rather than as model inputs. We further build a multi-task spatial instruction-tuning dataset (MTS) comprising 152{,}776 samples spanning 13 task types and three visual modalities, consolidated from six public datasets. Trained with a two-stage curriculum, GAMSI achieves state-of-the-art performance on seven spatial intelligence benchmarks.

URL PDF HTML 收藏
2605.24869 2026-05-26 cs.CL

Lngram: N-gram Conditional Memory in Latent Space

Lngram: 潜在空间中的N-gram条件记忆

Yunao Zheng, Guoyang Xia, Xiaojie Wang, Lei Ren

机构 * Beijing University of Posts and Telecommunications (BUPT)(北京邮电大学) Li Auto Inc.(Li Auto公司)

AI总结 提出Lngram,一种在潜在空间中学习离散符号并执行N-gram查找的条件记忆模块,以解耦检索与骨干网络,提升长上下文语言建模和跨模态任务性能。

详情
AI中文摘要

序列建模需要组合推理和局部静态知识检索,而标准Transformer通过密集计算处理两者。Engram部分地将检索与骨干网络解耦,但其基于token的键仍依赖于文本分词和哈希压缩。我们提出Lngram,一种潜在空间中的条件记忆模块,直接从隐藏状态学习离散符号,并对这些符号执行N-gram查找。该设计消除了对分词器ID的依赖,并自然地扩展到非文本模态。在我们的评估设置中,Lngram优于Transformer和Engram基线,在长上下文语言建模中持续降低困惑度,并在事后添加到预训练模型时有效注入领域知识。与骨干网络的联合训练进一步超越了完全微调,而在视觉-语言和视觉-语言-动作任务上的实验显示了整体提升。使用LogitLens和CKA的分析表明,Lngram使预测相关信息更早出现,在有限的推理和内存开销下增加了有效深度。代码可在https://github.com/zyaaa-ux/Lngram获取。

英文摘要

Sequence modeling requires both compositional reasoning and local static knowledge retrieval, yet standard Transformers handle both through dense computation. Engram partially decouples retrieval from the backbone, but its token-based keys remain tied to text tokenization and hash compression. We propose Lngram, a latent-space conditional memory module that learns discrete symbols directly from hidden states and performs N-gram lookup over these symbols. This design removes the dependence on tokenizer IDs and naturally extends to non-text modalities. In our evaluated settings, Lngram outperforms Transformer and Engram baselines, consistently reduces perplexity in long-context language modeling, and effectively injects domain knowledge when added post hoc to pretrained models. Joint training with the backbone further surpasses full fine-tuning, while experiments on vision-language and vision-language-action tasks show overall gains. Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier, increasing effective depth with limited inference and memory overhead. Code is available at https://github.com/zyaaa-ux/Lngram.

URL PDF HTML 收藏
2510.22973 2026-05-26 cs.CV

Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method

扩展以占据为中心的驾驶场景生成:数据集与方法

Bohan Li, Xin Jin, Hu Zhu, Hongsi Liu, Ruikai Li, Jiazhe Guo, Kaiwen Cai, Chao Ma, Yueming Jin, Hao Zhao, Xiaokang Yang, Wenjun Zeng

机构 * Shanghai Jiao Tong University(上海交通大学) Eastern Institute of Technology(东部技术研究院) School of Electronic Information and Electrical Engineering(电子信息与电气工程学院) Li Auto(力汽车) National University of Singapore(新加坡国立大学) Tsinghua University(清华大学) Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生实验室) Ningbo Institute of Digital Twin(宁波数字孪生研究院)

AI总结 针对占据数据稀缺问题,构建最大语义占据数据集Nuplan-Occ,并提出统一框架联合生成高质量语义占据、多视角视频和LiDAR点云,采用时空解耦架构及高斯泼溅稀疏点图渲染和传感器感知嵌入策略,实现高保真生成。

Comments IEEE TPAMI

详情
AI中文摘要

驾驶场景生成是自动驾驶的关键领域,支持下游应用,包括感知和规划评估。以占据为中心的方法通过提供跨帧和模态的一致条件,最近取得了最先进的结果;然而,其性能严重依赖于标注的占据数据,而这类数据仍然稀缺。为克服这一限制,我们整理了Nuplan-Occ,这是迄今为止最大的语义占据数据集,基于广泛使用的Nuplan基准构建。其规模和多样性不仅促进了大规模生成建模,也促进了自动驾驶下游应用。基于该数据集,我们开发了一个统一框架,联合合成高质量语义占据、多视角视频和LiDAR点云。我们的方法采用时空解耦架构,支持4D动态占据的高保真空间扩展和时间预测。为弥合模态差距,我们进一步提出了两种新技术:基于高斯泼溅的稀疏点图渲染策略,增强多视角视频生成;以及传感器感知嵌入策略,显式建模LiDAR传感器属性以实现逼真的多LiDAR模拟。大量实验表明,与现有方法相比,我们的方法实现了更优的生成保真度和可扩展性,并验证了其在下游任务中的实用价值。仓库:https://github.com/Arlo0o/UniScene-Unified-Occupancy-centric-Driving-Scene-Generation/tree/v2

英文摘要

Driving scene generation is a critical domain for autonomous driving, enabling downstream applications, including perception and planning evaluation. Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities; however, their performance heavily depends on annotated occupancy data, which still remains scarce. To overcome this limitation, we curate Nuplan-Occ, the largest semantic occupancy dataset to date, constructed from the widely used Nuplan benchmark. Its scale and diversity facilitate not only large-scale generative modeling but also autonomous driving downstream applications. Based on this dataset, we develop a unified framework that jointly synthesizes high-quality semantic occupancy, multi-view videos, and LiDAR point clouds. Our approach incorporates a spatio-temporal disentangled architecture to support high-fidelity spatial expansion and temporal forecasting of 4D dynamic occupancy. To bridge modal gaps, we further propose two novel techniques: a Gaussian splatting-based sparse point map rendering strategy that enhances multi-view video generation, and a sensor-aware embedding strategy that explicitly models LiDAR sensor properties for realistic multi-LiDAR simulation. Extensive experiments demonstrate that our method achieves superior generation fidelity and scalability compared to existing approaches, and validates its practical value in downstream tasks. Repo: https://github.com/Arlo0o/UniScene-Unified-Occupancy-centric-Driving-Scene-Generation/tree/v2

URL PDF HTML 收藏
2601.04537 2026-05-22 cs.LG cs.CL

Linear Dynamics in the RLVR Training of Large Language Models

在大语言模型RLVR训练中的线性动力学

Tianle Wang, Jiayu Liu, Zhongyuan Wu, Shenghao Jin, Wei Chen, Hao Xu, Ning Miao

机构 * Department of Data Science, City University of Hong Kong(香港城市大学数据科学系) Hong Kong Institute of AI for Science, City University of Hong Kong(香港城市大学人工智能科学研究院) Li Auto Inc. Beihang University(北航大学)

AI总结 本文研究了强化学习可验证奖励(RLVR)在大语言模型训练中的内部动态,发现RLVR在多种模型和训练配置下均进入线性区域,通过实验和理论分析证明这种线性特性源于训练信号的高方差和噪声,且具有预测性和实用性。

Comments Major revision: substantially reorganized the manuscript and added a theoretical explanation section. The replacement is intended for the same arXiv paper; the core topic and contribution remain the same

详情
AI中文摘要

强化学习可验证奖励(RLVR)在以推理为导向的大语言模型(LLMs)中推动了显著的性能提升,但其内部训练动态仍 largely 是一个黑箱。在本文中,我们对RLVR进行了全面的轨迹级分析,并揭示出一个显著的规律:在各种模型家族、RL算法和训练配置下,RLVR始终进入一个稳健的线性区域,其中参数权重和输出对数概率,通过严格教师强制评估测量,以高度线性的方式(R²>0.7)演变。通过受控实验和理论分析,我们证明这种线性并非偶然,而是源于RLVR训练信号的高方差和噪声性质,这些性质起到了低通滤波器的作用,将优化集中在稳定的、低维的漂移上。此外,我们显示这种线性结构不仅具有描述性,而且具有强大的预测性和实用性。具体而言,权重空间外推在性能上与标准RL优化相当,同时通过定期重新定位实现了6.1倍的训练加速。同时,输出空间外推作为一种轻量级干预,有效 bypassed 后期模型崩溃,持续在数学和编码基准上优于标准RL,平均性能提升了4.2%。我们的代码可在https://github.com/Miaow-Lab/RLVR-Linearity获得。

英文摘要

Reinforcement learning with verifiable rewards (RLVR) has driven significant performance gains in reasoning-oriented large language models (LLMs), yet its internal training dynamics remain largely a black box. In this work, we perform a comprehensive trajectory-level analysis of RLVR and uncover a striking regularity: across various model families, RL algorithms, and training configurations, RLVR consistently enters a robust linear regime, where both parameter weights and output log-probabilities, measured rigorously via teacher-forced evaluation, evolve in a highly linear manner ($R^2 > 0.7$). Through controlled experiments and theoretical analysis, we demonstrate that this linearity is not a coincidence, but stems from the high-variance, noisy nature of RLVR training signals, which act as a low-pass filter to concentrate optimization along a stable, low-dimensional drift. Moreover, we show that this linear structure is not merely descriptive but powerfully predictive and actionable. Specifically, weight-space extrapolation matches the performance of standard RL optimization while achieving a 6.1x training speedup through periodic re-grounding. Meanwhile, output-space extrapolation serves as a lightweight intervention that effectively bypasses late-stage model collapse, consistently outperforming standard RL across mathematical and coding benchmarks, with an average performance improvement of 4.2%. Our code is available at https://github.com/Miaow-Lab/RLVR-Linearity.

URL PDF HTML 收藏
2605.21372 2026-05-21 cs.CV cs.AI cs.LG cs.RO

Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Training

闭环动态驾驶数据混合用于真实-合成协同训练

Hongzhi Ruan, Pei Liu, Weiliang Ma, Zhengning Li, Xueyang Zhang, Jun Ma, Dan Xu, Kun Zhan

机构 * Li Auto(力汽车) HKUST(香港科技大学) HKUST (GZ)(香港科技大学(广州))

AI总结 本文提出了一种闭环动态数据混合方法,通过动态优化过程调整训练数据混合比例,以提升模型性能,解决了在有限预算下优化数据混合的关键问题。

详情
AI中文摘要

数据扩展是现代深度学习的基础,随着自动驾驶转向端到端学习,其重要性日益增加。现实世界驾驶数据标注成本高且场景偏向性明显,使利用几乎无限的合成数据进行真实-合成协同训练成为有前景的方向。然而,简单地整合所有可用的合成数据效率低下且导致分布偏移,优化实际训练预算下的数据混合仍是一个关键但尚未充分研究的问题。因此,我们主张在场景类型和数量上为训练数据混合提供明确指导。特别是在本文中,我们将数据混合近似概念化为一个动态优化过程,通过闭环评估反馈迭代调整训练数据混合以最大化模型性能,并提出AutoScale,一种完全自动化的闭环数据引擎,统一了场景表示、数据混合优化与检索以及模型训练与评估。具体而言,我们提出了图正则化的自编码器(Graph-RAE)用于驾驶场景表示,引入了簇感知梯度上升(Cluster-GA)用于簇级重要性估计和重新加权,并执行簇引导的向量检索以选择高价值样本。在NavSim上的实验表明,AutoScale在有限预算下优于传统协同训练和跨域基线,实现了更好的性能。

英文摘要

Data scaling is fundamental to modern deep learning, and grows increasingly critical as autonomous driving shifts to end-to-end learning. Real-world driving data is expensive to annotate and scene-biased, making real-synthetic co-training with near-infinite synthetic data a promising direction. However, naively incorporating all available synthetic data is inefficient and leads to distribution shifts, and optimizing data mixture under practical training budgets remains a critical yet under-explored problem. In this sense, we claim that the mixture of training data requires clear guidance in terms of scene types and quantities. Particularly in this work, we conceptualize the data mixture approximately as a dynamic optimization process that iteratively adjusts the training data mixture to maximize model performance, guided by closed-loop evaluation feedback, and propose AutoScale, a fully automated closed-loop data engine unifying scene representation, data mixture optimization and retrieval, as well as model training and evaluation. Specifically, we propose Graph Regularized AutoEncoder (Graph-RAE) for driving scene representations, introduce Cluster-aware Gradient Ascent (Cluster-GA) for cluster-wise importance estimation and reweighting, and perform cluster-guided vector retrieval to select high-value samples. Experiments on NavSim demonstrate that AutoScale outperforms vanilla co-training and cross-domain baselines, achieving better performance with fewer synthetic samples under constrained budgets.

URL PDF HTML 收藏
2605.20961 2026-05-21 cs.CV

Preserve, Reveal, Expand: Faithful 4D Video Editing with Region-Aware Conditioning

保留、揭示、扩展:基于区域感知的4D视频编辑

Zhangchi Hu, Wenzhang Sun, Xiangchen Yin, Jiahui Yuan, Chunfeng Wang, Hao Li, Kun Zhan, Xiaoyan Sun

机构 * University of Science and Technology of China(中国科学技术大学) Li Auto Inc.(利汽车公司)

AI总结 本文提出PREX框架,通过区域感知分解目标时空体积,解决4D视频编辑中区域保持、揭示和扩展的问题,提升了视频编辑的准确性和稳定性。

Comments 23 pages, 13 figures

详情
AI中文摘要

现有的4D驱动视频扩散模型主要针对合理生成,但忠实的4D编辑需要在合成遮挡或视外内容时保留源观测区域。我们识别出证据角色不匹配问题:可靠的源支持证据、不可靠的渲染提示和不支持的区域在单一条件信号中交织,导致保留漂移、鬼影和不稳定的外推。我们提出PREX(保留、揭示、扩展),一个区域感知框架,根据观测支持和场景范围将目标时空体积分解为保留、揭示和扩展角色。PREX通过校准置信度构建观测支持的外观提示,并通过区域感知适配器注入到冻结的视频扩散骨干网络中,通过代理任务训练而无需配对编辑视频。我们进一步引入PREBench,一个诊断基准,包含精心编辑、区域角色掩码和人类对齐的指标,补充了全局视频质量和4D控制评估。实验表明,PREX在减少区域结构失败的同时,保持了强大的视觉质量和4D编辑控制能力。项目页面:https://ricepastem.github.io/PREX-Open

英文摘要

Existing 4D-driven video diffusion models primarily target plausible generation, but faithful 4D editing requires preserving source-observed regions while synthesizing disoccluded or out-of-view content. We identify Evidence-Role Mismatch: reliable source-backed evidence, unreliable rendered cues, and unsupported regions are entangled in a single conditioning signal, causing preservation drift, ghosting, and unstable extrapolation. We propose PREX (Preserve, Reveal, Expand), a region-aware framework that decomposes the target spatiotemporal volume into Preserve, Reveal, and Expand roles according to observation support and scene extent. PREX builds observation-backed appearance cues with calibrated confidence and injects them into a frozen video diffusion backbone through a region-aware adapter, trained with proxy tasks without requiring paired edited videos. We further introduce PREBench, a diagnostic benchmark with curated edits, region-role masks, and human-aligned metrics that complement global video-quality and 4D-control evaluations. Experiments show that PREX reduces region-structured failures while maintaining strong visual quality and 4D edit control capability. Project Page: https://ricepastem.github.io/PREX-Open

URL PDF HTML 收藏
2605.18548 2026-05-19 cs.CL cs.AI

STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics

STT-Arena:一种更现实的工具使用环境,包含时空动态

Tingfeng Hui, Hao Xu, Pengyu Zhu, Hongsheng Xin, Kun Zhan, Sen Su, Chunxiao Liu, Ning Miao

机构 * Hong Kong Institute of AI for Science, City University of Hong Kong(香港人工智能科学研究院,香港城市大学) Department of Data Science, City University of Hong Kong(数据科学系,香港城市大学) Beijing University of Posts and Telecommunications(北京邮电大学) Li Auto Inc.(李汽有限公司) Independent Researcher(独立研究者)

AI总结 本文提出STT-Arena基准测试,旨在评估大型语言模型在面对时空动态变化时的适应性规划能力,发现现有模型在处理此类动态问题时存在显著不足,并提出改进方法STT-Agent-4B以提升性能。

Comments Work in progress

详情
AI中文摘要

大型语言模型(LLMs)在现实世界中的代理应用中必须能够重新规划和适应,当任务中途中断时推翻其先前决策。现有的动态基准主要测量LLMs是否能够及时检测时间变化,留下适应时空动态的互补挑战未被探索。我们介绍了STT-Arena(Spatio-Temporal Tool-Use Arena),一个包含227个高质量交互任务的基准测试,涵盖九种时空冲突类型和四种可解性级别。每个任务都基于一个现实、可执行的环境,配备注入的时空触发器,可以突然使正在进行的计划失效,迫使模型检测状态变化并构建修订的执行策略。对前沿LLMs的广泛评估显示,即使是最先进的专有模型,如Claude-4.6-Opus,也只达到低于40%的总体准确率,突显了时空动态推理的根本难度。对失败轨迹的系统分析揭示了现有模型的三种反复出现的错误模式:停滞状态执行、动态触发器的误诊断和缺失的适应后验证。基于这些发现,我们提出了一种迭代轨迹细化技术,消除这些失败模式,结合在线强化学习,产生STT-Agent-4B,其在STT-Arena上优于前沿LLMs。

英文摘要

Large language models (LLMs) deployed in real-world agentic applications must be capable of replanning and adapting when mid-task disruptions invalidate their prior decisions. Existing dynamic benchmarks primarily measure whether LLMs can detect temporal changes in a timely manner, leaving the complementary challenge of adaptive replanning under spatio-temporal dynamics largely unexplored. We introduce STT-Arena (Spatio-Temporal Tool-Use Arena), a benchmark of 227 high-quality interactive tasks spanning nine spatio-temporal conflict types and four solvability levels. Each task is grounded in a realistic, executable environment equipped with injected spatio-temporal triggers that can abruptly invalidate an ongoing plan, forcing the model to detect the state shift and construct a revised execution strategy. Extensive evaluation of frontier LLMs reveals that even the SOTA proprietary models, including Claude-4.6-Opus, achieves less than 40\% overall accuracies, highlighting the fundamental difficulty of spatio-temporal dynamic reasoning. Systematic analysis of failure trajectories uncovers three recurring error modes of existing models: Stale-State Execution, Misdiagnosis of Dynamic Triggers, and Missing Post-Adaptation Verification. Guided by these findings, we propose an iterative trajectory refinement technique that eliminates these failure patterns from training data, and combine it with online RL to produce STT-Agent-4B which outperforms frontier LLMs on STT-Arena.

URL PDF HTML 收藏
2605.12625 2026-05-15 cs.RO cs.CV

Driving Intents Amplify Planning-Oriented Reinforcement Learning

意图驱动强化学习放大规划导向的强化学习

Hengtong Lu, Victor Shea-Jay Huang, Chengmin Yang, Pengfei Jing, Jifeng Dai, Yan Xie, Benjin Zhu

机构 * Li Auto(力 auto) Tsinghua University(清华大学) CUHK MMLab(香港大学MMLab)

AI总结 本文提出DIAL框架,通过两阶段方法解决连续动作策略在偏好对齐中的模式崩溃问题,通过意图引导的采样扩展分布并提升性能。

Comments Project page: https://mind-omni.github.io/

详情
AI中文摘要

连续动作策略在单个演示轨迹上训练时会出现模式崩溃:样本集中在演示操作周围,策略无法表示语义不同的替代方案。在基于偏好的评估中,这限制了最佳N性能——即使使用oracle选择也无法恢复采样分布不包含的内容。我们引入DIAL,一种两阶段驾驶意图放大强化学习框架,用于偏好对齐的连续动作驾驶策略。第一阶段,DIAL将流匹配动作头条件于离散意图标签上,使用分类器无关引导(CFG),扩展采样分布沿不同的操作模式,打破单演示模式崩溃。第二阶段,DIAL将扩展的分布通过多意图GRPO带入偏好强化学习,涵盖每个偏好组内的所有意图类别,防止微调重新崩溃到当前偏好模式。实例化为端到端驾驶,使用八个规则推导的意图,并在WOD-E2E: 竞争性视觉到动作(VA)和视觉语言到动作(VLA)监督微调(SFT)基线中评估。基线在最佳128下低于人类驾驶演示,最强先验(RAP)在最佳64下最高达到评分8.5;意图-CFG采样将此上限提升到RFS 9.14,首次超越先前最佳(RAP 8.5)和人类驾驶演示(8.13);多意图GRPO将保留RFS从7.681提升到8.211,而每个单一意图基线峰值较低且训练后退化。这些结果表明,偏好强化学习在连续动作策略上训练的瓶颈不仅在于如何更新策略,还在于扩展和保留被优化的采样分布。

英文摘要

Continuous-action policies trained on a single demonstrated trajectory per scene suffer from mode collapse: samples cluster around the demonstrated maneuver and the policy cannot represent semantically distinct alternatives. Under preference-based evaluation, this caps best-of-N performance -- even oracle selection cannot recover what the sampling distribution does not contain. We introduce DIAL, a two-stage Driving-Intent-Amplified reinforcement Learning framework for preference-aligned continuous-action driving policies. In the first stage, DIAL conditions the flow-matching action head on a discrete intent label with classifier-free guidance (CFG), which expands the sampling distribution along distinct maneuver modes and breaks single-demonstration mode collapse. In the second stage, DIAL carries this expanded distribution into preference RL through multi-intent GRPO, which spans all intent classes within every preference group and prevents fine-tuning from re-collapsing around the currently preferred mode. Instantiated for end-to-end driving with eight rule-derived intents and evaluated on WOD-E2E: competitive Vision-to-Action (VA) and Vision-Language-Action (VLA) Supervised Finetuning (SFT) baselines plateau below the human-driven demonstration at best-of-128, with the strongest prior (RAP) capping at Rater Feedback Score (RFS) 8.5 even with best-of-64; intent-CFG sampling lifts this ceiling to RFS 9.14 at best-of-128, surpassing both the prior best (RAP 8.5) and the human-driven demonstration (8.13) for the first time; and multi-intent GRPO improves held-out RFS from 7.681 to 8.211, while every single-intent baseline peaks lower and degrades by training end. These results suggest that the bottleneck of preference RL on continuous-action policies trained from demonstrations is not only how to update the policy, but to expand and preserve the sampling distribution being optimized.

URL PDF HTML 收藏
2605.12624 2026-05-15 cs.RO cs.CV

MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving

MindVLA-U1:统一流架构使VLA超越VA用于自动驾驶

Yuzhou Huang, Benjin Zhu, Hengtong Lu, Victor Shea-Jay Huang, Haiming Zhang, Wei Chen, Jifeng Dai, Yan Xie, Hongsheng Li

机构 * CUHK MMLab(香港中文大学多模态实验室) Li Auto Tsinghua University(清华大学)

AI总结 MindVLA-U1通过统一流架构实现自动驾驶中VLA超越VA,采用统一视觉语言模型骨干和流匹配连续动作轨迹,在保持各模态自然输出形式的同时,实现高效规划和低延迟。

Comments Work in progress. Project page: https://mind-omni.github.io/

详情
AI中文摘要

自动驾驶从模块化流水线向端到端统一发展,视觉-语言-动作(VLA)模型是这一旅程的自然延伸。在实践中,驾驶VLA常在规划质量上落后于VA,表明问题不在于模型规模,而在于如何将语义推理、时间上下文和连续控制结合。我们主张这一差距反映了VLA的构建方式——作为孤立子任务改进,未能组成连贯的驾驶能力。我们提出了MindVLA-U1,首个用于自动驾驶的统一流VLA架构。统一的VLM骨干在单次前向传递中生成AR语言标记(可选)并流匹配连续动作轨迹,通过共享表示保持各模态的自然输出形式。完整的流设计按驱动视频帧处理而非固定视频-动作块,在成本高昂的时间VLM建模下。规划轨迹在帧间平滑演变,而学习的流内存通道承载时间上下文并更新。统一的架构通过灵活的自注意力上下文管理,使密集和稀疏MoT骨干实现快/慢系统,并暴露可测量的语言控制路径用于动作:语言预测的驾驶意图通过分类器无关引导(CFG)引导动作扩散,将语言侧意图转化为连续动作规划的控制信号。在长尾WOD-E2E基准上,MindVLA-U1首次超越经验人类驾驶员(8.20 RFS vs. 8.13 GT RFS)使用2步扩散,实现了比先前VA/VLA大得多的规划ADEs,并在保持自然语言接口用于人车交互的同时匹配VA延迟(16 FPS vs. RAP的18 FPS在1B规模下)

英文摘要

Autonomous driving has progressed from modular pipelines toward end-to-end unification, and Vision-Language-Action (VLA) models are a natural extension of this journey beyond Vision-to-Action (VA). In practice, driving VLAs have often trailed VA on planning quality, suggesting that the difficulty is not simply model scale but the interface through which semantic reasoning, temporal context, and continuous control are combined. We argue that this gap reflects how VLA has been built -- as isolated subtask improvements that fail to compose coherent driving capabilities -- rather than what VLA is. We present MindVLA-U1, the first unified streaming VLA architecture for autonomous driving. A unified VLM backbone produces AR language tokens (optional) and flow-matching continuous action trajectories in a single forward pass over one shared representation, preserving the natural output form of each modality. A full streaming design processes the driving video framewise rather than as fixed video-action chunks under costly temporal VLM modeling. Planned trajectories evolve smoothly across frames while a learned streaming memory channel carries temporal context and updates. The unified architecture enables fast/slow systems on dense & sparse MoT backbones via flexible self-attention context management, and exposes a measurable language-control path for action: language-predicted driving intents steers the action diffusion via classifier-free guidance (CFG), turning language-side intent into control signals for continuous action planning. On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time (8.20 RFS vs. 8.13 GT RFS) with 2 diffusion steps, achieves state-of-the-art planning ADEs over prior VA/VLA by large margins, and matches VA latency (16 FPS vs. RAP's 18 FPS at 1B scale) while preserving natural language interfaces for human-vehicle interaction.

URL PDF HTML 收藏
2605.12622 2026-05-15 cs.RO cs.CV

Action Emergence from Streaming Intent

从流式意图中涌现动作

Pengfei Jing, Victor Shea-Jay Huang, Hengtong Lu, Jifeng Dai, Yan Xie, Benjin Zhu

机构 * Li Auto Tsinghua University(清华大学) CUHK

AI总结 本文提出流式意图方法,通过连续思维链生成语义合适且安全的动作,实现自动驾驶中长尾交通场景的自主决策。

Comments Project page: https://mind-omni.github.io/

详情
AI中文摘要

我们正式将动作涌现定义为端到端自动驾驶的目标能力:通过场景条件推理生成物理可行、语义合适且安全合规的动作,而非通过检索或插值已学习的场景-动作映射。我们证明了先前范式无法实现动作涌现:自回归轨迹解码器将本质上多模态的未来压缩成单一平均输出,而扩散和流匹配生成器表达多模态但无法由推理意图控制。我们提出流式意图作为实现动作涌现的具体方法:一种机制使驾驶意图(i)通过连续思维链语义流推导自场景理解,(ii)在片段间时间流推导使意图承诺在驾驶时间线上保持一致。我们在名为SI(流式意图)的VLA模型中实现了流式意图。SI自回归解码四步思维链并发射意图标记;解码后的意图驱动分类器自由引导(CFG)在流匹配动作头部,仅需两步去噪生成最终轨迹。在Waymo端到端基准测试中,SI在验证集上获得RFS得分为7.96,在测试集上为7.74。除了汇总指标外,该模型展示了首次在完全端到端VLA中的意图忠实可控性:对于固定场景,推理时改变意图类别会产生质不同但一致高质量的计划,纯由数据驱动学习生成,无需任何预建轨迹库或手码后处理选择器。

英文摘要

We formalize action emergence as a target capability for end-to-end autonomous driving: the ability to generate physically feasible, semantically appropriate, and safety-compliant actions in arbitrary, long-tail traffic scenes through scene-conditioned reasoning rather than retrieval or interpolation of learned scene-action mappings. We show that previous paradigms cannot deliver action emergence: autoregressive trajectory decoders collapse the inherently multimodal future into a single averaged output, while diffusion and flow-matching generators express multimodality but are not steerable by reasoned intent. We propose Streaming Intent as a concrete way to approach action emergence: a mechanism that makes driving intent (i) semantically streamed through a continuous chain-of-thought that causally derives the intent from scene understanding, and (ii) temporally streamed across clips so that intent commitments remain coherent along the driving horizon. We realize Streaming Intent in a VLA model we call SI (Streaming Intent). SI autoregressively decodes a four-step chain-of-thought and emits an intent token; the decoded intent then drives classifier-free guidance (CFG) on a flow-matching action head, requiring only two denoising steps to generate the final trajectory. On the Waymo End-to-End benchmark, SI achieves competitive aggregate performance, with an RFS score of 7.96 on the validation set and 7.74 on the test set. Beyond aggregate metrics, the model demonstrates -- to our knowledge for the first time in a fully end-to-end VLA -- intent-faithful controllability: for a fixed scene, varying the intent class at inference yields qualitatively distinct yet consistently high-quality plans, arising purely from data-driven learning without any pre-built trajectory bank or hand-coded post-hoc selector.

URL PDF HTML 收藏
2605.12163 2026-05-14 cs.CV

Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model

自洽潜在推理:面向视觉语言模型的长潜在序列推理

Chenfeng Wang, Wei He, Xuhan Zhu, Chunpeng Zhou, Qizhen Li, Song Yan, Yufei Zheng, Chengjun Yu, Fan Lu, Wei Zhai, Yang Cao, Pengfei Yu, Zheng-Jun Zha

机构 * University of Science and Technology of China(中国科学技术大学) Li Auto Inc.(利亚自动化公司)

AI总结 本文提出SCOLAR模型,通过轻量级detransformer生成辅助视觉token,结合三阶段SFT和ALPO强化学习,显著提升视觉语言模型的长序列推理能力,达到开源模型在现实任务中的最佳性能。

Comments 17 pages, 6 figures

详情
AI中文摘要

在语言推理中,更长的推理链通常能获得更好的性能,这自然暗示视觉潜在推理可能也受益于更长的潜在序列。然而,我们发现一个反直觉现象:现有潜在视觉推理方法的性能随着潜在序列变长而系统性下降。我们揭示了根本原因:信息增益崩溃——自回归生成使每一步高度依赖于先前输出,因此后续token几乎无法引入新信息。我们进一步发现,用于监督的重采样(≥128×)图像嵌入提供的信号量并不比无意义占位符多。受这些见解启发,我们提出SCOLAR(Self-COnsistent LAtent Reasoning),引入轻量级detransformer,利用LLM的完整序列隐藏状态生成辅助视觉token,每个token独立锚定于原始视觉空间。结合三阶段SFT和ALPO强化学习,SCOLAR将可接受的潜在CoT长度扩展了超过30倍,实现了在现实任务基准上的开源模型最佳性能(比基础模型高出14.12%),并展示了强大的分布外泛化能力。

英文摘要

In language reasoning, longer chains of thought consistently yield better performance, which naturally suggests that visual latent reasoning may likewise benefit from longer latent sequences. However, we discover a counterintuitive phenomenon: the performance of existing latent visual reasoning methods systematically degrades as the latent sequence grows longer. We reveal the root cause: Information Gain Collapse -- autoregressive generation makes each step highly dependent on prior outputs, so subsequent tokens can barely introduce new information. We further identify that heavily pooled ($\geq 128\times$) image embeddings used as supervision targets provide no more signal than meaningless placeholders. Motivated by these insights, we propose SCOLAR (Self-COnsistent LAtent Reasoning), which introduces a lightweight detransformer that leverages the LLM's full-sequence hidden states to generate auxiliary visual tokens in a single shot, with each token independently anchored to the original visual space. Combined with three-stage SFT and ALPO reinforcement learning, SCOLAR extends acceptable latent CoT length by over $30\times$, achieves state-of-the-art among open-source models on real-world reasoning benchmarks (+14.12% over backbone), and demonstrates strong out-of-distribution generalization.

URL PDF HTML 收藏