arXivDaily arXiv每日学术速递 周一至周五更新

高校专区

Harbin Institute of Technology(哈尔滨工业大学)

至 收录 1398
2605.04445 2026-07-21 cs.CV 版本更新

LEGO: LoRA-Enabled Generator-Oriented Framework for Synthetic Image Detection

LEGO:用于合成图像检测的基于LoRA的面向生成器框架

Yutong Xiao, Ran Ran, Jiwei Wei, Shuchang Zhou, Ke Liu, Zheng Ziqiang, Caiyan Qin

机构 * University of Electronic Science and Technology of China(电子科技大学) Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))

AI总结 针对生成技术使合成图像难辨真伪,现有检测方法存在泛化和过拟合问题,提出LEGO框架。通过MLP调制多个LoRA模块,分两阶段训练,能提取特定生成器独特伪影,在少数据和少轮次训练下性能优于现有方法。

Comments 10 pages,2 figures

详情
AI中文摘要

生成技术的快速发展使得合成图像几乎与真实图像难以区分,因此迫切需要强大的检测器来应对错误信息。现有方法主要依赖于多个生成器共享的通用伪影特征,但随着生成器多样性增加,这些共同特征的重叠逐渐减少,严重损害模型泛化能力,而仅关注独特伪影又易导致过拟合。为此提出LEGO框架,其核心机制是用MLP调制多个LoRA模块,各模块预训练以捕获特定生成器的独特伪影,再进行基于注意力的特征融合。LEGO将训练过程分为两个阶段,把独特伪影提取委托给专门的LoRA模块。每个LoRA模块在单生成器数据集上单独训练以学习特定生成器表示,然后MLP和注意力层在混合数据集上训练以动态调节各模块贡献。受益于模块化且强大的设计,LEGO可通过纳入新的LoRA模块自然扩展以适应新出现的下一代数据集,在训练图像数量少、训练数据占比低且训练轮次少的情况下,仍比现有最优方法性能大幅提升。

英文摘要

The rapid advancement of generative technologies has made synthetic images nearly indistinguishable from real ones, thereby creating an urgent need for robust detectors to counter misinformation. However, existing methods mainly rely on universal artifact features that are shared across multiple generators. We observe that as the diversity of generators increases, the overlap of these common features gradually decreases. This severely undermines model generalization. In contrast, focusing only on unique artifacts tends to cause overfitting to specific forgery patterns. To address this challenge, we propose LEGO (LoRA-Enabled Generator-Oriented Framework). The core mechanism of LEGO employs an MLP to modulate multiple LoRA (Low-Rank Adaptation) blocks, each pretrained to capture the unique artifacts of a specific generator, followed by attention-based feature fusion. Unlike conventional methods that seek a single universal solution, LEGO delegates unique artifact extraction to specialized LoRA modules by dividing its training procedure into two stages. Each LoRA module is individually trained on a single-generator dataset to learn generator-specific representations, then MLP and attention layers are trained on mixed datasets to dynamically regulate the contribution of each module. Benefiting from its modular yet robust design, LEGO can be naturally extended by incorporating new LoRA modules for adaptation to newly emerging next-generation datasets, while still achieving substantially better performance than prior SOTA methods with fewer than 30,000 training images, less than 10% of their training data, and only 5 epochs in each training stage.

URL PDF HTML 收藏
2607.18046 2026-07-21 cs.LG 新提交

SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

SEE:用于长视野GUI代理轨迹合成的结构感知探索与利用

Zhuohang Fan, Beichen Zhang, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang

机构 * Harbin Institute of Technology, Weihai(哈尔滨工业大学(威海)) Harbin Institute of Technology (Weihai) Qingdao Research Institute(哈尔滨工业大学(威海)青岛研究院)

AI总结 针对GUI代理轨迹合成中缺乏高覆盖率长视野轨迹的问题,提出SEE两阶段数据合成框架,通过高效探索和基于图的合成,产生可重复可解释数据,避免虚假循环,提升代理任务成功率和泛化能力,还将发布代码和数据集。

Comments Accepted by ACM International Conference on Multimedia 2026 (ACM MM 2026)

详情
AI中文摘要

由视觉语言模型驱动的图形用户界面(GUI)代理有望实现现实世界移动任务的自动化。然而,进展受到从元素丰富且快速发展的应用程序中收集的高覆盖率、长视野交互轨迹缺乏的限制。现有管道通常依赖昂贵的人工演示或策略框架,容易过度采样常见流程,而遗漏罕见转换和复杂多步程序。为解决此问题,我们提出SEE,一个两阶段数据合成框架,包括一个高效探索阶段,在屏幕和元素上构建显式UI转换图,以及一个基于图的合成阶段,通过规划和受控采样组合不同的多步轨迹。这种设计产生可重复和可解释的数据生成,同时明确防止虚假循环并实现长视野组合。在多个真实世界应用程序中,SEE产生平均长度为14.8步的轨迹,同时避免虚假循环,在SEE上微调的代理实现了更高的任务成功率和对未见屏幕的泛化能力。我们将公开发布合成代码和数据集。

英文摘要

Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps. Existing pipelines often rely on costly human demonstrations or on-policy framework, which tends to over-sample common flows while missing rare transitions and complex multi-step procedures. To address this problem, we propose SEE, a two-stage data synthesis framework consisting of (i) an efficient exploration stage that builds an explicit UI transition graph over screens and elements, and (ii) a graph-based synthesis stage that composes diverse multi-step trajectories via planning and controlled sampling. This design yields reproducible and explainable data generation, while explicitly preventing spurious cycles and enabling long-horizon composition. Across multiple real-world apps, SEE produces trajectories with an average length of 14.8 steps while avoiding spurious loops, and agents fine-tuned on SEE achieve improved task success and generalization to unseen screens. We will publicly release our synthesis code and dataset.

URL PDF HTML 收藏
2607.17896 2026-07-21 cs.CV 新提交

Locality-Aware Density Control for Efficient Gaussian-based Image Representation

用于高效基于高斯的图像表示的局部感知密度控制

Jiacong Chen, Qingyu Mao, Xiandong Meng, Shuai Liu, Chao Li, Fanyang Meng, Youneng Bao, Yongsheng Liang

机构 * Shenzhen University(深圳大学) Pengcheng Laboratory(鹏城实验室) Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))

AI总结 研究二维高斯喷溅图像表示中高斯容量分配低效问题,提出局部感知密度控制框架LocoADC,通过区域高斯致密化和相似性驱动高斯合并策略及局部颜色一致性约束,有效提升高斯容量利用率,改进图像表示。

Comments Accepted by ACMMM 2026

详情
AI中文摘要

二维高斯喷溅因其明确的公式、快速的光栅化和良好的解码效率,是图像表示的一个有吸引力的方向。该范式的表示质量取决于高斯容量在需求区域的适当分配。现有方法在优化过程中未能有效分配高斯容量。我们提出了局部感知密度控制(LocoADC),一个即插即用的框架,通过区域高斯致密化(RGD)和相似性驱动高斯合并(SDGM)策略提高高斯容量利用率,还有局部颜色一致性约束实现更可靠的合并。实验表明LocoADC通过更有效的局部高斯分配持续改进多个基线。

英文摘要

2D Gaussian Splatting is an attractive direction for image representation due to its explicit formulation, fast rasterization, and favorable decoding efficiency. The representation quality of this paradigm depends on the proper allocation of Gaussian capacity to the demanding regions. However, existing methods fail to allocate Gaussian capacity efficiently during optimization: under-reconstructed content is often refined in a fragmented pixel-wise manner, while neighboring optimized Gaussians with similar attributes are redundantly retained. This inefficiency motivates the need for a density control framework that jointly addresses insufficient allocation in under-reconstructed regions and redundant allocation in over-reconstructed regions. Our key insight is that this framework should exploit two complementary forms of locality: the local continuity of reconstruction errors in image space for improved Gaussian allocation, and the local similarity of neighboring Gaussians in Gaussian space for redundant elimination. Based on this insight, we propose Locality-Aware Density Control (LocoADC), a plug-and-play framework that improves Gaussian capacity utilization through Region-wise Gaussian Densification (RGD) and Similarity-Driven Gaussian Merging (SDGM) strategies, together with a local color consistency constraint for more reliable merging. Extensive experiments on diverse datasets show that LocoADC consistently improves multiple baselines by enabling more effective local Gaussian allocation, including a 2.93 dB PSNR gain over GI on the CLIC dataset under the same 30k Gaussian budget. Code is available at: \textit{https://github.com/ChenJiaCong-1005/LocoADC}.

URL PDF HTML 收藏
2607.17861 2026-07-21 cs.RO cs.AI 新提交

ConceptTree: Bringing Semantic Transparency to Black-Box Decision Making for Robotic Manipulation

ConceptTree:为机器人操作的黑箱决策带来语义透明度

Yongyan Wen, Feifan Liu, Jinyi Chen, Bo An, Peng Liu, Siyuan Li

机构 * Harbin Institute of Technology(哈尔滨工业大学) Nanyang Technological University(南洋理工大学)

AI总结 研究长期机器人操作中可解释决策问题,提出ConceptTree框架,将高级技能选择重构为基于视觉观察的概念推理。该方法学习归一化概念空间并训练决策树,实验表明其优于现有基线,支持细粒度干预以纠正决策错误。

详情
AI中文摘要

在长期的机器人操作中建立可解释的决策过程对于实现可靠的人类监督和干预至关重要。然而,现有的机器人操作方法大多将技能选择视为从观察到行动的不透明映射,对决策形成方式的透明度有限。在这项工作中,我们提出了ConceptTree,一个将高级操作技能选择重新构建为对人类可解释概念进行推理的框架,将高级策略表示为基于视觉观察的一系列概念级谓词。我们的方法不是依赖于隐式潜在表示,而是学习一个基于视觉输入的归一化概念空间,在其上训练决策树以预测高级技能。这种表述产生了一个可追溯和可干预的透明决策过程,能够直接检查和修改策略行为。我们在一组复杂度不断增加的实际机器人操作任务上评估了我们的方法。实验结果表明,ConceptTree始终优于现有的基于概念的基线,特别是在复杂的长期场景中。此外,我们提供的定性案例研究表明,我们的模型通过修改单个概念支持细粒度干预,能够在不重新训练的情况下有针对性地纠正决策错误。

英文摘要

Establishing interpretable decision-making processes in long-horizon robotic manipulation is critical for enabling reliable human oversight and intervention. However, existing approaches to robotic manipulation largely treat skill selection as opaque mappings from observations to actions, offering limited transparency into how decisions are formed. In this work, we propose ConceptTree, a framework that reframes high-level manipulation skill selection as reasoning over human-interpretable concepts, representing high-level policies as a sequence of concept-level predicates over visual observations. Rather than relying on implicit latent representations, our method learns a normalized concept space grounded in visual inputs, over which a decision tree is trained to predict high-level skills. This formulation yields a transparent decision process that is both traceable and intervenable, enabling direct inspection and modification of policy behavior. We evaluate our approach on a set of real-world robotic manipulation tasks with increasing complexity. Experimental results show that ConceptTree consistently outperforms existing concept-based baselines, particularly in complex, long-horizon scenarios. Furthermore, we provide qualitative case studies showing that our model supports fine-grained intervention by modifying individual concepts, enabling targeted correction of decision errors without retraining.

URL PDF HTML 收藏
2607.17675 2026-07-21 cs.CV 新提交

ShotPlan: Cinematic Video Generation with Learnable Planning Token

ShotPlan:使用可学习规划令牌生成电影级视频

Su Guo, Guangce Liu, Haosen Yang, Jiepeng Wang, Cong Liu, Junqi Liu, Haibin Huang, Hongxun Yao, Chi Zhang, Xuelong Li

机构 * Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI)) Harbin Institute of Technology(哈尔滨工业大学)

AI总结 针对电影级视频生成难题,提出ShotPlan框架,基于视频扩散基础模型,引入可学习规划令牌捕捉镜头过渡线索,结合分数时间旋转位置嵌入在帧级别建模,实验证明其优于现有方法,提升了镜头管理与一致性。

Comments Project page: https://pensioner-11.github.io/ShotPlan/

详情
AI中文摘要

当前视频生成模型在单镜头生成方面成果显著,但在电影级视频生成中存在局限,因其需要连贯叙事和有效的多镜头构图,这需要明确的镜头规划。为应对这一挑战,我们提出ShotPlan,这是一个基于视频扩散基础模型的用于明确多镜头电影级视频生成的框架。我们的方法引入了可学习的规划令牌,它能捕捉镜头级过渡线索,并可与原始视频生成令牌无缝集成以控制过渡时间戳。与标准视频生成令牌不同,所提出的规划令牌配备了分数时间旋转位置嵌入(FRoPE),使镜头过渡能在帧级别建模。实验表明,ShotPlan显著优于现有的电影级视频生成方法,提供了更灵活的镜头管理和更强的镜头间一致性。

英文摘要

Current video generation models achieve impressive results in single-shot generation, yet remain limited in cinematic video generation, where coherent narratives and effective multi-shot composition require explicit shot planning. To address this challenge, we propose ShotPlan, a framework for explicit multi-shot cinematic video generation built upon a video diffusion foundation model. Our method introduces learnable planning tokens that capture shot-level transition cues and can be seamlessly integrated with the original video generation tokens to control transition timestamps. Unlike standard video generation tokens, the proposed planning tokens are equipped with Fractional Temporal Rotary Position Embedding (FRoPE), enabling shot transitions to be modeled at the frame level. Experiments demonstrate that ShotPlan significantly outperforms existing cinematic video generation methods, offering more flexible shot management and stronger inter-shot consistency.

URL PDF HTML 收藏
2607.17651 2026-07-21 cs.RO 新提交

HCPG-Flow:Hierarchical Contact-Progress Guidance for Flow-Policy Robot Manipulation

HCPG-Flow:用于流策略机器人操作的分层接触进展引导

Guanghu Xie, Mingxu Li, Shuo Zhang, Yonglong Zhang, Yifan Yang, Yang Liu, Zongwu Xie, Baoshi Cao

机构 * State Key Laboratory of Robotics and Systems, Harbin Institute of Technology(机器人技术与系统国家重点实验室,哈尔滨工业大学)

AI总结 研究机器人操作流策略中动作选择问题,提出HCPG-Flow方法,通过分层、以对象为中心的接触进展引导增强SAC-Flow,在模拟和物理任务中提高成功率,减少完成时间。

详情
AI中文摘要

流策略可表示机器人操作的多模态动作分布,但机器人在每个控制步骤必须执行一个动作。当采样多个提议时,基于评论家的排序使数据收集依赖于对重放中可能表示较弱的候选动作的值估计。我们引入了HCPG-Flow,这是一种分析性的展开时间选择器,它在保留其演员和评论家目标的同时,用分层的、以对象为中心的接触进展引导增强了SAC-Flow。HCPG在接触后从末端执行器方法切换到任务进展,通过与任务相关距离的一阶减少对每个提议进行评分,在候选集中标准化分数,并执行温度控制的动作嵌入。在十个模拟任务中,HCPG在两个基准上均提高了SAC-Flow的平均成功率,包括在Maniskill上提高了9.5个百分点。四个物理任务进一步显示出高成功率,成功完成时间减少了17.4%。

英文摘要

Flow policies can represent multimodal action distributions for robot manipulation, yet a robot must execute one action at each control step. When several proposals are sampled, critic-based ranking makes data collection depend on value estimates over candidate actions that may be weakly represented in replay. We introduce HCPG-Flow, an analytic rollout-time selector that augments SAC-Flow with hierarchical, object-centric contact-progress guidance while preserving its actor and critic objectives. HCPG switches from end-effector approach to task progress after contact, scores each proposal by the first-order reduction of a task-relevant distance, standardizes scores within the candidate set, and executes a temperature-controlled action embedding. Across ten simulated tasks, HCPG improves mean success over SAC-Flow on both benchmarks, including a 9.5 percentage-point gain on Maniskill. Four physical tasks further show high success with a 17.4% reduction in successful completion steps.Project page: https://hitxraz.github.io/HCPG-Flow/

URL PDF HTML 收藏
2607.17340 2026-07-21 cs.CV 新提交

Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection

用于域增量目标检测的正交知识刷新

Aoting Zhang, Dongbao Yang, Chang Liu, Xiaopeng Hong, Can Ma, Yu Zhou

机构 * Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所) Nankai University(南开大学) School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院) Tsinghua University(清华大学) Harbin Institute of Technology(哈尔滨工业大学)

AI总结 研究域增量目标检测问题,提出正交知识刷新(OKR)框架,通过构建特定域子空间、采用正交刷新策略和拓扑感知一致性,减少知识干扰和语义碎片化,实验表明该方法在mAP上比最佳无范例方法有显著提升。

Comments Accepted by ECCV 2026

详情
AI中文摘要

域增量目标检测(DIOD)要求模型在保留先验知识的同时不断适应新域。参数高效微调虽有前景,但存在覆盖关键过去知识、引发域间干扰和性能下降的风险。我们提出正交知识刷新(OKR)框架,通过为每个域构建独立的特定域子空间并融合进行整体决策,还提出基于梯度的正交刷新策略及拓扑感知一致性来减少干扰和语义碎片化。实验验证了OKR的优越性,在Pascal VOC和BDD100K系列上分别比最佳无范例方法的mAP高出5.6%和6.5%。

英文摘要

Domain-incremental object detection (DIOD) requires models to continually adapt to new domains while preserving prior knowledge. Recently, parameter-efficient fine-tuning offers a promising avenue, wherein a pre-trained model is frozen and a small number of learnable parameters are injected for downstream tasks. However, these methods risk overwriting critical past knowledge, triggering inter-domain interference and performance degradation. To address this challenge, we propose Orthogonal Knowledge Refreshing (OKR), a simple yet effective framework for DIOD. OKR incrementally constructs independent domain-specific subspaces via dedicated low-rank branches for each domain, which are seamlessly fused for a holistic decision, enabling conflict-free capacity expansion without domain selection during inference. To minimize knowledge interference during fusion, we present a gradient-based orthogonal refreshing strategy that projects gradient updates of new domains onto the orthogonal complement of the fused historical subspace, supporting continual adaptation without forgetting. Moreover, to mitigate semantic fragmentation across domains, we enforce topology-aware consistency, aligning the semantic structures of old and new domains. Extensive experiments validate the superiority of OKR, outperforming the best exemplar-free method by significant margins of +5.6% and +6.5% mAP on the Pascal VOC and BDD100K series, respectively.

URL PDF HTML 收藏
2607.17247 2026-07-21 cs.LG cs.AI 新提交

Distilled Reinforcement Learning for LLM Post-training

用于大语言模型训练后处理的蒸馏强化学习

Chen Wang, Zhaochun Li, Jionghao Bai, Yining Zhang, Hexuan Deng, Ge Lan, Yue Wang

机构 * College of Elite Engineers, Nankai University(南开大学精英工程师学院) Zhongguancun Academy(中关村学院) Beijing Institute of Technology(北京理工大学) Zhejiang University(浙江大学) Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所) Harbin Institute of Technology(哈尔滨工业大学) College of Software, Nankai University(南开大学软件学院)

AI总结 研究大语言模型训练后处理问题,提出蒸馏强化学习方法,集成教师监督到RL目标,含反向重要性采样等三个组件,能有效转移知识,在家族内和跨家族蒸馏实验中性能大幅优于标准RL和OPD。

详情
AI中文摘要

大语言模型训练后处理对于提升推理、适应性和对齐性至关重要。现有方法主要有强化学习(RL)和策略蒸馏(OPD)两种范式。RL依赖粗粒度结果监督,存在信用分配困难和获取新知识能力有限的问题。OPD通过KL散度无条件匹配教师逻辑,导致类似教师提供新知识少,差异大的教师指导无效,限制其用于家族内蒸馏。我们提出蒸馏强化学习(Distilled RL),将教师监督集成到RL目标中,提供细粒度指导,选择性转移新知识并避免无条件模仿。它包含反向重要性采样、负样本重置和序列级几何归一化三个组件。通过案例研究表明其能有效转移知识,实验显示在家族内和跨家族蒸馏设置中,Distilled RL在pass@1和pass@k方面均大幅优于标准RL和OPD。

英文摘要

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.

URL PDF HTML 收藏
2607.16768 2026-07-21 cs.LG 新提交

Robust Losses from Univariate Base Functions for Noisy-Label Learning

用于噪声标签学习的单变量基函数鲁棒损失

Peng Hu, Jianwei Ma

机构 * School of Mathematics, Harbin Institute of Technology(哈尔滨工业大学数学学院) School of Earth and Space Sciences, Peking University(北京大学地球与空间科学学院)

AI总结 针对噪声标签学习问题,提出从单变量基函数构建鲁棒多类损失的框架,开发两种方案并分析其性质,推导理论标准,所提损失在多噪声设置下经实验验证有竞争力或更优。

详情
AI中文摘要

有噪声标签学习是训练可靠深度神经网络的基本问题。鲁棒损失函数是减轻标签噪声负面影响的直接有效方法。但现有多数鲁棒损失直接针对最终多类目标设计,难以系统表征和扩展其鲁棒性。本文提出从单变量基函数构建鲁棒多类损失的通用框架,通过定义映射算子,可根据基函数简单性质表征诱导损失的鲁棒性。开发了两种互补构建方案,分析了其对称和不对称性质并推导充分条件,为噪声鲁棒损失设计提供理论标准。该框架还为构建对称损失提供新途径。在合成和真实世界噪声标签基准上的大量实验表明,所提损失在各种噪声设置下具有竞争力或更优性能。

英文摘要

Learning with noisy labels is a fundamental problem in training reliable deep neural networks. Robust loss functions provide a direct and effective way to mitigate the adverse effects of label noise. However, most existing robust losses are designed directly at the level of the final multiclass objective, which makes it difficult to systematically characterize and extend their robustness properties. In this paper, we propose a general framework that constructs robust multiclass losses from univariate base functions. By defining mapping operators from base functions to multiclass losses, the robustness of the induced losses can be characterized through simple properties of the base functions. We develop two complementary construction schemes, Target Separation and Binary Reduction, corresponding to inter-class independent and inter-class dependent formulations, respectively. For both schemes, we analyze their symmetry and asymmetry properties and derive corresponding sufficient conditions, which provide theoretical criteria for noise-robust loss design. The proposed framework also provides a new route to constructing symmetric losses, serving as a complement to normalization-based symmetric loss designs. Extensive experiments on synthetic and real-world noisy-label benchmarks demonstrate that the proposed losses achieve competitive or superior performance under various noise settings.

URL PDF HTML 收藏
2607.16327 2026-07-21 cs.CV 新提交

Localization-Infused Vision-Language Semantic Fusion for Text-Guided Medical Image Segmentation

用于文本引导医学图像分割的定位注入视觉语言语义融合

Songyue Han, Mingye Zou, Shuchang Ye, Lei Bi, Mingyuan Meng

机构 * Air Force Engineering University(空军工程大学) Harbin Institute of Technology(哈尔滨工业大学) University of Sydney(悉尼大学) Institute of Translational Medicine, Shanghai Jiao Tong University(上海交通大学转化医学研究院) Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院)

AI总结 研究针对文本引导医学图像分割,提出LoG框架,通过联合执行多尺度目标定位任务,实现三级定位注入语义融合,在三个基准数据集实验中表现出色,优于现有医学图像分割方法。

详情
AI中文摘要

医学图像分割对现代计算机辅助医学至关重要。近期,文本引导分割通过纳入临床医生制定的文本报告作为图像分割的语义指导展现出前景。现有方法通常通过预训练文本编码器隐式提取文本语义,再经简单图像 - 文本特征融合整合视觉 - 语言语义。但这些方法未明确捕捉文本报告中面向目标的信息,也未探索基础特征级融合之外的多级信息融合策略。本研究提出LoG框架,通过联合执行多尺度目标定位任务,明确捕捉面向目标的视觉 - 语言语义,实现三级定位注入语义融合,包括定位引导特征融合、定位门控注意力融合和定位约束损失融合。在三个基准数据集上的大量实验表明,LoG在QaTa - COV19、MosMedData + 和Kvasir - SEG上分别取得91.59%、80.71%和94.59%的Dice分数,持续优于现有方法。

英文摘要

Medical image segmentation is essential for modern computer-aided medicine. Recently, text-guided segmentation has shown promise by incorporating clinician-formulated textual reports as semantic guidance for image segmentation. These reports describe target appearance, location, and neighboring anatomy, providing explicit guidance for localization and delineation. Existing text-guided segmentation methods typically extract textual semantics implicitly through a pretrained text encoder and then integrate vision-language semantics via straightforward image-text feature fusion. However, these methods do not explicitly capture target-oriented information embedded in textual reports, particularly target location, and do not explore multi-level information fusion strategies beyond basic feature-level fusion, limiting the extraction and integration of critical textual semantics. In this study, we propose LoG, a localization-infused vision-language fusion framework for text-guided medical image segmentation. By jointly performing multi-scale target localization tasks, LoG explicitly captures target-oriented vision-language semantics and enables three-level localization-infused semantic fusion: (i) localization-guided feature fusion that directly infuses location-relevant semantics into visual features, (ii) localization-gated attention fusion that redirects multi-scale localization predictions to reinforce critical regions, and (iii) localization-constrained loss fusion that supervises segmentation based on spatial consistency with target localization. Extensive experiments on three benchmark datasets, involving three medical imaging modalities with paired textual reports, demonstrate that LoG achieves Dice scores of 91.59%, 80.71%, and 94.59% on QaTa-COV19, MosMedData+, and Kvasir-SEG, respectively, consistently outperforming state-of-the-art medical image segmentation methods.

URL PDF HTML 收藏
2607.16322 2026-07-21 cs.CV cs.AI 新提交

GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs

GMoT:用于基于多模态大语言模型的细粒度微手势视频推理的门控运动感知令牌化

Taorui Wang, Wei Xia, Hui Ma, Zijia Song, Jiayu Zhang, Zeheng Wang, Yong Xu, Zitong Yu

机构 * Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)) Great Bay University(大湾区大学) Dongguan Key Laboratory for Intelligence and Information Technology(东莞市智能信息技术重点实验室)

AI总结 研究针对微手势识别中运动易被掩盖及MLLMs处理运动学困难的问题,提出GMoT模块,通过空间加权池等提取运动线索并融合,引入渐进奖励引导策略优化范式,在多数据集上表现出色,还提出BRG召回及跨域转移协议。

Comments Accepted to ACM MM 2026

详情
AI中文摘要

微手势识别需要检测短暂的、空间局部化的运动,这些运动经常被占主导地位的静态外观和背景噪声所掩盖。虽然多模态大语言模型(MLLMs)在一般视频理解方面表现出色,但它们在处理微妙的运动学方面存在固有困难,并且通常依赖于静态姿势先验。为此,我们提出了GMoT,一种门控运动感知令牌化模块,它在时间建模之前将稀疏的运动学证据明确提炼成一个紧凑的序列。GMoT通过空间加权池动态突出与动作相关的区域,提取相邻帧的时间差分以捕获精确的运动能量,并使用保守初始化的语义门将这些线索自适应地融合到视觉流中。为了从简单分类过渡到基于证据的推理,我们进一步引入了一种渐进奖励引导的策略优化范式,由一个生成解剖学重点字幕的半监督注释管道支持。除了在iMiGUE(67.32%)和SMG(73.11%)上的比较方法中实现最佳的Top-1准确率,将Qwen3-VL-8B基线提高了+6.80和+3.11个百分点外,我们的框架还引入了基于正确预测的身体区域定位(BRG)召回作为解剖学定位代理,以及iMiGUE和SMG之间的重叠标签跨域转移协议。广泛的评估表明,我们的GMoT增强模型提高了域内准确率,在标签保留损坏下保持了明显的收益,并在明确的小分割条件下提高了面向准确率的跨域转移,同时在其生成的理由中保持了高解剖学定位。

英文摘要

Micro-gesture recognition demands the detection of fleeting, spatially localized movements that are frequently overwhelmed by dominant static appearances and background noise. While Multimodal Large Language Models (MLLMs) excel at general video understanding, they inherently struggle with subtle kinematics and often rely on static posture priors. To this end, we propose GMoT, a Gated Motion-Aware Tokenization module that explicitly distills sparse kinematic evidence into a compact sequence prior to temporal modeling. GMoT dynamically spotlights action-relevant regions via spatially weighted pooling, extracts adjacent-frame temporal differencing to capture precise motion energy, and adaptively fuses these cues into the visual stream using a conservatively initialized semantic gate. To transition from simple classification to evidence-grounded reasoning, we further introduce a progressive reward-guided policy refinement paradigm, supported by a semi-supervised annotation pipeline that generates anatomically focused captions. Beyond achieving the best Top-1 accuracy among the compared methods on iMiGUE (67.32\%) and SMG (73.11\%), improving the Qwen3-VL-8B baseline by +6.80 and +3.11 points, our framework introduces Body-Region Grounding (BRG) Recall as an anatomical-grounding proxy conditioned on correct predictions, together with an overlapping-label cross-domain transfer protocol between iMiGUE and SMG. Extensive evaluations demonstrate that our GMoT-augmented model improves in-domain accuracy, retains clear gains under label-preserving corruptions, and improves accuracy-oriented cross-domain transfer under explicit small-split caveats while maintaining high anatomical grounding in its generated rationales.

URL PDF HTML 收藏
2607.16244 2026-07-21 cs.LG cs.AI cs.CL 新提交

CIGPO: Contextual Information-Gain Policy Optimization for Multi-Turn Evidence-Reading LLM Agents

CIGPO:用于多轮证据阅读语言模型代理的上下文信息增益策略优化

Hao Dou

机构 * Harbin Institute of Technology(哈尔滨工业大学)

AI总结 研究多轮证据阅读语言模型代理训练不稳定问题,提出CIGPO方法,通过方差注入策略,利用冻结参考模型对数似然边际增加作逐轮信号,避免奖励方差崩溃,在HotpotQA实验中取得更好效果。

详情
AI中文摘要

仅通过结果进行强化学习来训练多轮证据阅读代理是不稳定的,因为中间轮次获得的直接奖励很少。在Qwen2.5 - 3B - Instruct的HotpotQA实验中,GRPO最初有所改进(标准F1为0.430),但随后崩溃为100%格式违规输出。训练日志诊断揭示了零优势锁定机制。我们提出了一种方差注入策略,通过为中间证据阅读轮次分配逐轮奖励来防止奖励分布崩溃。上下文信息增益策略优化(CIGPO)使用冻结参考模型对真实答案的对数似然的边际增加作为逐轮信号来实现此策略。通过IG和F1奖励的单独归一化以及IG权重课程,CIGPO在3B规模的HotpotQA上达到了标准F1为0.518,而最佳GRPO检查点为0.430,最终GRPO检查点为0.000。CIGPO在整个训练过程中保持有意义的奖励方差并避免零优势锁定。这些结果表明奖励方差崩溃是仅结果GRPO的一种具体失败模式,并且轮级IG奖励可以在HotpotQA设置中防止这种情况。

英文摘要

Training multi-turn evidence-reading agents with outcome-only reinforcement learning is unstable because intermediate turns receive little direct credit. In HotpotQA experiments with Qwen2.5-3B-Instruct, GRPO initially improves (standard F1 0.430) but subsequently collapses to 100% format-violating outputs. Training-log diagnosis reveals a zero-advantage lock-in mechanism: all sampled trajectories receive the minimum format penalty (-2.0), group-relative advantages vanish, and the policy-gradient loss becomes zero--an optimization deadlock. We propose a variance-injection strategy: by assigning per-turn rewards to intermediate evidence-reading turns, we prevent the group reward distribution from collapsing to a single value--preserving the variation that GRPO's group-relative advantage requires. Contextual Information-Gain Policy Optimization (CIGPO) implements this strategy using the marginal increase in the frozen reference model's log-likelihood of the ground-truth answer as the per-turn signal. With separate normalization of IG and F1 rewards and an IG-weight curriculum, CIGPO reaches a standard F1 of 0.518 on HotpotQA at the 3B scale (from 0.252 base; +105%), compared with 0.430 for the best GRPO checkpoint and 0.000 for the final GRPO checkpoint. CIGPO maintains meaningful reward variance and avoids zero-advantage lock-in throughout training. These results identify reward-variance collapse as a concrete failure mode of outcome-only GRPO and show that turn-level IG rewards can prevent it in this HotpotQA setting.

URL PDF HTML 收藏
2605.24651 2026-07-21 math.NA cs.LG cs.NA 版本更新

WINO: A Weak-Form Physics Informed Neural Operator for Hyperelasticity on Variable Domains

WINO: 一种用于变域超弹性问题的弱形式物理信息神经算子

Bokai Zhu, Yizheng Wang, Qinghui Zhang, Timon Rabczuk

机构 * School of Science, Harbin Institute of Technology, Shenzhen, P. R. China(哈尔滨工业大学深圳校区) School of Science, Harbin Institute of Technology, Shenzhen, Guangdong(哈尔滨工业大学深圳校区) Institute of Structural Mechanics, Bauhaus-Universität Weimar(魏玛 Bauhaus 大学结构力学研究所)

AI总结 提出一种无数据框架WINO,结合神经算子的效率与φ-有限元法的几何灵活性,通过最小化弱形式残差和惩罚项训练,实现高精度且计算时间减少50-80%。

详情
AI中文摘要

我们提出了一种弱形式物理信息神经算子(WINO),这是一个无数据框架,结合了神经算子的效率与φ-有限元法(φ-FEM)的几何灵活性。φ-FEM是一种非拟合方法,无需体拟合网格即可适应几何变化,其中域几何由水平集函数φ表示。为了施加边界条件,Dirichlet问题采用φ-FEM提升,因此仅学习齐次位移贡献,而牵引驱动的Neumann问题额外预测非拟合弱形式所需的辅助场。参数通过最小化与φ-FEM对齐的弱形式残差平方以及切割单元辅助方程的平方惩罚来训练,从而消除了对大型配对数据集的依赖。训练后,WINO输出可作为神经算子热启动(NOWS)为非线性φ-FEM求解器提供初始值,相比传统冷启动求解器减少了迭代次数。数值基准测试表明,WINO在所有基准测试中实现了低于0.04的高精度,同时与纯数据驱动方法相比,总计算时间减少了50-80%。

英文摘要

We propose a Weak-form Physics-Informed Neural Operator (WINO), a data-free framework that combines the efficiency of neural operators with the geometric flexibility of the $φ$-finite element method ($φ$-FEM). $φ$-FEM is an unfitted method that accommodates geometric variations without body-fitted meshes, where the domain geometry is represented by the level-set function $φ$. To impose the boundary conditions, Dirichlet problems adopt the $φ$-FEM lifting so only the homogeneous displacement contribution is learned, whereas traction-driven Neumann problems additionally predict the auxiliary fields necessary for the unfitted weak formulation. Parameters are trained by minimizing squared weak-form residuals aligned with $φ$-FEM together with squared penalties on the cut-cell auxiliary equations, which removes the need for large paired datasets of converged reference solutions. When labeled reference data are available, an optional data-augmented variant (WINO+data) can further combine this physics-informed loss with a supervised term. After training, WINO outputs can seed the nonlinear $φ$-FEM solvers as neural operator warm starts (NOWS), which reduce iteration counts relative to traditional cold-started solvers. Numerical benchmarks show substantial accuracy of WINO together with total training times of about 15%-70% of those of supervised $φ$-FEM-FNO across all cases, without requiring reference-solution generation.

URL PDF HTML 收藏
2602.01167 2026-07-21 cs.AI

Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models

所有个体层都有帮助吗?视觉-语言模型中任务干扰层的实证研究

Zhiming Liu, Yujie Wei, Lei Feng, Xiu Su, Xiaobo Xia, Weili Guan, Zeke Xie, Shuo Yang

机构 * Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)) Harbin Institute of Technology(哈尔滨工业大学) Southeast University(东南大学) Central South University(中南大学) National University of Singapore(新加坡国立大学) The Hong Kong University of Science and Technology, Guangzhou(香港科学与技术大学(广州))

AI总结 研究通过层干预发现部分层阻碍下游任务,提出任务自适应层剔除方法提升性能,揭示预训练VLM的意外模块化特性。

Journal ref Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, 2026, pp. 9597-9607

详情
AI中文摘要

当前VLM在多种多模态任务中表现出色,但默认启用所有层可能阻碍任务表现。通过干预单层参数发现,某些层反而抑制任务性能。系统研究各层对不同任务的影响,提出任务-层交互向量量化方法,并引入无需训练的测试时适应方法TaLo,动态剔除最干扰的层,提升模型在多个任务和数据集上的性能,包括提升Qwen-VL在ScienceQA地图任务上的准确率。

英文摘要

Current VLMs have demonstrated capabilities across a wide range of multimodal tasks. Typically, in a pretrained VLM, all layers are engaged by default to make predictions on downstream tasks. We find that intervening on a single layer, such as by zeroing its parameters, can improve the performance on certain tasks, indicating that some layers hinder rather than help downstream tasks. We systematically investigate how individual layers influence different tasks via layer intervention. Specifically, we measure the change in performance relative to the base model after intervening on each layer and observe improvements when bypassing specific layers. This improvement can be generalizable across models and datasets, indicating the presence of Task-Interfering Layers that harm downstream tasks' performance. We introduce Task-Layer Interaction Vector, which quantifies the effect of intervening on each layer of a VLM given a task. These task-interfering layers exhibit task-specific sensitivity patterns: tasks requiring similar capabilities show consistent response trends under layer interventions, as evidenced by the high similarity in their task-layer interaction vectors. Inspired by these findings, we propose TaLo (Task-Adaptive Layer Knockout), a training-free, test-time adaptation method that dynamically identifies and bypasses the most interfering layer for a given task. Without parameter updates, TaLo improves performance across various models and datasets, including boosting Qwen-VL's accuracy on the Maps task in ScienceQA by up to 16.6%. Our work reveals an unexpected form of modularity in pretrained VLMs and provides a plug-and-play, training-free mechanism to unlock hidden capabilities at inference time. The source code will be publicly available.

URL PDF HTML 收藏
2603.16461 2026-07-21 cs.CV 版本更新

GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models

GAP-MLLM:几何对齐预训练以激活多模态大语言模型中的3D空间感知

Jiaxin Zhang, Junjun Jiang, Haijie Li, Youyu Chen, Kui Jiang, Dave Zhenyu Chen

机构 * Harbin Institute of Technology(哈尔滨工业大学) School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院) Huawei(华为)

AI总结 本文提出GAP-MLLM,通过几何对齐预训练激活多模态大语言模型中的3D空间感知,改进了传统方法在3D空间感知上的不足。

Comments Accepted by ECCV 2026. Project page: https://gapmllm.github.io/

详情
AI中文摘要

多模态大语言模型(MLLMs)在语义推理方面表现出色,但在纯RGB输入限制下难以实现3D空间感知。尽管利用了3D重建模型的隐式几何先验,基于图像的方法在性能上仍逊于使用显式3D数据的方法。我们认为,这种差距并非源于几何先验不足,而是训练范式不匹配:以文本为主的微调未能激活MLLMs中的几何表示。现有方法通常采用简单的特征拼接并直接优化下游任务,缺乏几何特定的监督。为此,我们提出GAP-MLLM,一种几何对齐的预训练范式,旨在在下游适应前显式激活结构感知。具体而言,我们引入了一个视觉提示的联合任务,迫使MLLMs预测稀疏点云图并同时预测语义标签,从而强制几何意识。此外,我们设计了多级渐进融合模块,具有令牌级门控机制,使几何先验能够自适应地整合而不抑制语义推理。大量实验表明,GAP-MLLM显著增强了几何特征融合,并在3D视觉定位、3D密集标注和3D视频目标检测任务中持续提升了性能。

英文摘要

Multimodal Large Language Models (MLLMs) demonstrate exceptional semantic reasoning but struggle with 3D spatial perception when restricted to pure RGB inputs. Despite leveraging implicit geometric priors from 3D reconstruction models, image-based methods still exhibit a notable performance gap compared to methods using explicit 3D data. We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm: text-dominated fine-tuning fails to activate geometric representations within MLLMs. Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision, leading to suboptimal structural utilization. To address this limitation, we propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation. Specifically, we introduce a visual-prompted joint task that compels the MLLMs to predict sparse pointmaps alongside semantic labels, thereby enforcing geometric awareness. Furthermore, we design a multi-level progressive fusion module with a token-level gating mechanism, enabling adaptive integration of geometric priors without suppressing semantic reasoning. Extensive experiments demonstrate that GAP-MLLM significantly enhances geometric feature fusion and consistently enhances performance across 3D visual grounding, 3D dense captioning, and 3D video object detection tasks.

URL PDF HTML 收藏
2507.20804 2026-07-21 cs.AI 版本更新

MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

MMGraphRAG: 通过可解释的多模态知识图谱弥合视觉与语言的鸿沟

Xueyao Wan, Hang Yu

机构 * Harbin Institute of Technology(哈尔滨工业大学) Nanyang Technological University(南洋理工大学)

AI总结 MMGraphRAG通过引入可解释的多模态知识图谱,解决多模态场景下的实体链接和跨模态推理问题,实现最先进的多模态信息处理性能。

详情
AI中文摘要

大型语言模型(LLMs)常常出现幻觉问题,检索增强生成(RAG)和图RAG通过引入外部知识和知识图谱(KGs)来缓解这一问题。然而,图RAG仍然以文本为中心,因为构建细粒度的多模态知识图谱(MMKGs)困难重重。现有的融合方法,如共享嵌入或描述,需要任务特定的训练,并未能保留视觉结构知识或跨模态推理路径。为弥合这一差距,我们提出了MMGraphRAG,通过一种新颖的跨模态融合方法将视觉场景图与文本知识图谱相结合。它引入了SpecLink,一种利用谱聚类进行准确的跨模态实体链接和基于路径的检索的方法,以引导生成。我们还发布了CMEL数据集,专门设计用于复杂多模态场景中的细粒度多实体对齐。在CMEL、DocBench和MMLongBench上的评估表明,MMGraphRAG实现了最先进的性能,展示了强大的领域适应能力和优越的多模态信息处理能力。

英文摘要

Large Language Models (LLMs) suffer from hallucinations due to their static parametric knowledge. Retrieval-Augmented Generation (RAG) and GraphRAG mitigate this issue by incorporating external knowledge and structured reasoning over knowledge graphs (KGs). However, existing approaches remain largely text-centric, as constructing fine-grained multimodal knowledge graphs (MMKGs) with explicit cross-modal semantics remains challenging. In this paper, we propose MMGraphRAG, a framework for building interpretable MMKGs that unify textual and visual knowledge. Our approach represents visual content as structured scene graphs and integrates them with textual KGs through a novel cross-modal entity linking method, SpecLink, which leverages spectral clustering to jointly model semantic similarity and graph structure. This design preserves explicit entities, relations, and reasoning paths across modalities, enabling structure-aware retrieval and generation. To support evaluation, we introduce the CMEL dataset, a benchmark for fine-grained cross-modal entity alignment. Experimental results on CMEL demonstrate improved entity linking accuracy, while evaluations on DocBench and MMLongBench show that MMGraphRAG achieves superior performance and stronger robustness, particularly in complex multimodal reasoning scenarios.

URL PDF HTML 收藏
2607.15713 2026-07-20 eess.SP cs.AI cs.LG 新提交

Map as a Prompt: Learning Multi-Modal Spatial-Signal Foundation Models for Cross-scenario Wireless Localization

地图作为提示:学习用于跨场景无线定位的多模态空间信号基础模型

Yong Chu, Xun Zhou, Zenglin Xu, Hui Wang, Yue Yu

机构 * Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)) Pengcheng Laboratory(鹏城实验室) Shanghai Academy of AI for Science(上海人工智能科学研究院) Artificial Intelligence Innovation and Incubation Institute, Fudan University(复旦大学人工智能创新与孵化院)

AI总结 针对无线定位在不同环境中面临的挑战及现有方法的局限,提出多模态基础模型SigMap,通过循环自适应掩码策略和“地图即提示”框架学习无线表示并实现跨场景适应,实验证明其性能领先且零样本泛化能力强。

Comments 17pages, 9 figures, poster in International Conference on Learning Representations (ICLR), 2026

详情
AI中文摘要

准确且稳健的无线定位是新兴5G/6G应用(如自动驾驶、扩展现实和智能制造)的关键推动因素。尽管其很重要,但由于无线信号的复杂性及其对环境变化的敏感性,在不同环境中实现精确的定位仍然具有挑战性。现有的数据驱动方法通常泛化能力有限,需要大量标记数据且难以适应新场景。为解决这些限制,我们提出了SigMap,这是一种多模态基础模型,引入了两个关键创新:(1)一种循环自适应掩码策略,基于信道周期性特征动态调整掩码模式以学习稳健的无线表示;(2)一种新颖的“地图即提示”框架,通过轻量级软提示集成3D地理信息以实现有效的跨场景适应。大量实验表明,我们的模型在多个定位任务中实现了领先的性能,在未见环境中展现出强大的零样本泛化能力,显著优于有监督和自监督的基线。

英文摘要

Accurate and robust wireless localization is a critical enabler for emerging 5G/6G applications, including autonomous driving, extended reality, and smart manufacturing. Despite its importance, achieving precise localization across diverse environments remains challenging due to the complex nature of wireless signals and their sensitivity to environmental changes. Existing data-driven approaches often suffer from limited generalization capability, requiring extensive labeled data and struggling to adapt to new scenarios. To address these limitations, we propose SigMap, a multimodal foundation model that introduces two key innovations: (1) A cycle-adaptive masking strategy that dynamically adjusts masking patterns based on channel periodicity characteristics to learn robust wireless representations; (2) A novel "map-as-prompt" framework that integrates 3D geographic information through lightweight soft prompts for effective cross-scenario adaptation. Extensive experiments demonstrate that our model achieves state-of-the-art performance across multiple localization tasks while exhibiting strong zero-shot generalization in unseen environments, significantly outperforming both supervised and self-supervised baselines by considerable margins.

URL PDF HTML 收藏
2605.15677 2026-07-20 cs.CL cs.CV 版本更新

VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing

VCG-Bench:迈向统一的视觉导向基准,用于结构化生成与编辑

Xiaoyan Su, Peijie Dong, Zhenheng Tang, Song Tang, Yuyao Zhai, Kaitao Lin, Liang Chen, Gai Yuhang, Yuyu Luo, Qiang Wang, Xiaowen Chu

机构 * The Hong Kong University of Science and Technology (GuangZhou)(香港科学与技术大学(广州)) Huawei Technologies Co., Ltd(华为技术有限公司) Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)) South China University of Technology(华南理工大学)

AI总结 本文提出VCG-Bench,一个统一的视觉导向mxGraph任务基准,通过符号逻辑和XML实现精确的图表生成与编辑,解决现有方法在结构化任务中的局限性。

Comments Accepted by ICML2026, 37 pages, 10 figures

详情
AI中文摘要

尽管视觉语言模型(VLMs)迅速发展,但在处理专业工作流程中至关重要的结构化、可控图表任务方面仍存在关键差距。现有方法主要依赖像素级合成,其在可编辑性和保真度上存在固有限制。本文提出一种新的图表即代码范式,利用mxGraph可扩展标记语言(XML)进行精确的图表生成与编辑。我们提出了VCG-Bench,一个统一的视觉导向mxGraph任务基准。VCG-Bench包括:(1)一个包含1,449种不同图表的分类数据集,涵盖6个领域和15个子领域;(2)一种整合生成(视觉到代码)和可编辑性(代码到代码)的范式定义;(3)一种定制的评估协议,采用多维指标,如mxGraph执行成功率、风格一致性分数(SCS)等。实验结果突显了当前最先进(SOTA)VLMs在结构保真度和指令合规性方面的挑战,反映了其视觉和推理能力。

英文摘要

Despite the rapid advancements in Vision-Language Models (VLMs), a critical gap remains in their ability to handle structured, controllable diagrammatic tasks essential for professional workflows. Existing methods predominantly rely on pixel-based synthesis, which operates in probabilistic pixel spaces and is inherently limited in editability and fidelity. Instead, we propose a new Diagram-as-Code paradigm with symbolic logic that leverages mxGraph Extensible Markup Language (XML) for precise diagram generation and editing. We present VCG-Bench, a unified benchmark for visual-centric \texttt{mxGraph} tasks. VCG-Bench comprises: (1) a taxonomized dataset of 1,449 diverse diagrams spanning 6 domains and 15 sub-domains, (2) a paradigm definition that integrates Generation (Vision-to-Code) and Editability (Code-to-Code), (3) a Tailored Evaluation Protocol employing multi-dimensional metrics such as \texttt{mxGraph} Execution Success Rate, Style Consistency Score (SCS), etc. Experimental results highlight the challenges faced by current State-of-the-Art (SOTA) VLMs in structured fidelity and instruction compliance, reflecting their vision and reasoning capabilities.

URL PDF HTML 收藏
2604.06742 2026-07-20 cs.SE cs.AI 版本更新

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

评估基于LLM的从零开始软件生成在端到端CLI工具场景中的表现

Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, David Lo

机构 * Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳)) Independent Researcher, China(独立研究者,中国) Singapore Management University, Singapore(新加坡管理大学)

AI总结 本文提出CLI-Tool-Bench基准,用于评估从零生成CLI工具的能力,发现顶级模型成功率低于43%,指出生成代码倾向单体化。

Comments Data link: https://github.com/kinesiatricssxilm14/CLI-Tool-Bench

详情
AI中文摘要

大型语言模型(LLM)正在推动向意图驱动开发转变,其中代理从零构建完整软件。然而,现有基准无法评估这种从零开始生成能力,因为依赖预定义框架忽略仓库结构规划,且刚性白盒单元测试缺乏端到端行为验证。为弥合这一差距,我们引入CLI-Tool-Bench,一个结构无关的基准,用于评估从零生成CLI工具的能力。它包含100个多样化的现实仓库,通过黑盒差分测试框架评估。代理生成的软件在沙箱中执行,通过多层等价度度量比较系统副作用和终端输出与人工编写的结果。评估七种最先进的LLM后,发现顶级模型成功率低于43%,凸显了从零生成的持续挑战。此外,更高的token消耗并不保证更好的性能,且代理倾向于生成单体代码。

英文摘要

The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch. However, existing benchmarks fail to adequately assess this 0-to-1 generation capability due to two fundamental limitations. First, they rely on predefined structural scaffolds, which reduces the task to mere file-filling. Second, they depend on rigid white-box unit testing, which forces generated code to conform to specific internal implementations rather than validating end-to-end user-centric behavior. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structure-agnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by an automated black-box differential testing framework, the benchmark comprises 94 high-quality, real-world repositories spanning diverse programming languages and complexity levels. For each task, agents are provided with an empty workspace, forcing them to autonomously handle repository planning and dependencies. We evaluate the generated software by executing it in isolated sandboxes. The system-level side effects and terminal outputs are then compared against human-written oracles using a rigorous multi-tiered equivalence metric. Extensive evaluation of seven state-of-the-art LLMs reveals that the top-tier models achieve a maximum overall success rate of only 43.8%, highlighting that 0-to-1 software generation remains a highly challenging frontier. Furthermore, we discover that agents exhibit a strong tendency to generate monolithic code structures, and that higher token consumption does not necessarily yield better task performance.

URL PDF HTML 收藏
2607.14683 2026-07-17 cs.AI 新提交

InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring

InCarEmo:用于车内情感识别和驾驶员状态监测的多模态数据集

Hao Yang, Yanyan Zhao, Kewei Zhao, Hongbo Zhang, Tian Zheng, Yusheng Liu, Xing Fu, Bichen Wang, Yu Zhang, Hao He, Zhen Wu, Xuda Zhi, Yongbo Huang, Bing Qin

机构 * Harbin Institute of Technology(哈尔滨工业大学) SERES(赛力斯)

AI总结 针对现有车内情感计算数据集局限,引入多模态InCarEmo数据集,整合多种数据支持情感识别、疲劳检测和分心监测等任务,构建中英文基准并给出基线结果,证明多模态融合优势,为车内情感理解奠基,推动人车交互发展。

详情
AI中文摘要

理解驾驶员的情绪和状态对于确保安全并增强人车交互的下一代智能车内系统至关重要。然而,现有的车内情感计算公共数据集大多局限于视觉模态,很少包含对话信息,难以捕捉驾驶员情绪背后的语言和交互线索。为填补这些空白,我们引入了InCarEmo,一个用于车内情感识别和驾驶员状态监测的多模态数据集。它整合了RGB和红外视频、车内音频以及从模拟现实驾驶员行为的脚本化车内场景收集的对话文本,涵盖多种光照条件和驾驶环境。该数据集支持多模态情感识别、疲劳检测和分心监测这三项主要任务。除了原始中文数据,还构建了辅助英文基准以支持初步跨语言评估。我们提供了一个统一基准,给出了单模态和多模态方法的广泛基线结果,包括模态缺失和噪声条件下的分析。实验结果证明了多模态融合的好处,并揭示了现实世界噪声和低光条件下仍存在的挑战。通过发布InCarEmo,我们旨在为强大、可解释且以人为本的车内情感理解建立全面基础,促进更安全、更具同理心的人车交互。

英文摘要

Understanding driver emotion and state is critical for the next generation of intelligent in-cabin systems that ensure safety and enhance human-vehicle interaction. However, existing public datasets for in-cabin affective computing are largely limited to visual modalities and rarely include conversational information, making it difficult to capture the linguistic and interactive cues underlying driver emotion. To address these gaps, we introduce InCarEmo, a multimodal dataset for in-cabin emotion recognition and driver state monitoring. InCarEmo integrates RGB and infrared video, in-cabin audio, and dialogue text collected from scripted in-cabin scenarios designed to simulate realistic driver behaviors, covering diverse lighting conditions and driving contexts. The dataset supports three primary tasks: 1) multimodal emotion recognition, 2) fatigue detection, and 3) distraction monitoring. In addition to the original Chinese data, we construct an auxiliary English benchmark to support preliminary cross-lingual evaluation. We provide a unified benchmark with extensive baseline results across unimodal and multimodal methods, including analyses under modality-missing and noise conditions. Experimental results demonstrate the benefits of multimodal fusion and reveal remaining challenges under real-world noise and low-light conditions. By releasing InCarEmo, we aim to establish a comprehensive foundation for robust, interpretable, and human-centric in-cabin affective understanding, promoting safer and more empathetic driver-vehicle interaction.

URL PDF HTML 收藏
2601.19788 2026-07-17 cs.LG cs.DC 版本更新

Knowledge-Aware Evolution for Task-Free Streaming Federated Continual Learning with Arbitrary Class Overlap

具有类别重叠和无任务标识符的流式联邦持续学习中的知识感知进化

Sixing Tan, Xianmin Liu

机构 * Faculty of Computing(计算学院) Harbin Institute of Technology(哈尔滨工业大学)

AI总结 FedKACE通过自适应推理模型切换、梯度平衡重放和核谱边界缓冲区维护,解决流式联邦持续学习中类别重叠和无任务标识符带来的知识混淆问题。

详情
AI中文摘要

联邦持续学习(FCL)利用客户端之间的协作,在非平稳数据中平衡新知识的获取和先前知识的保留。然而,现有的批量基于FCL方法缺乏对流式场景的适应性,这些场景中旧数据和新数据之间存在类别重叠,并且没有任务标识符,导致旧知识和新知识难以区分,样本的任务分配不确定,知识混淆。为了解决这个问题,我们提出了流式联邦持续学习的设置:在每次联邦学习(FL)轮次中,客户端处理流式数据,这些数据具有不相交的样本和可能重叠的类别,且没有任务标识符,需要在每次FL轮次后持续保持对所有先前类别的推理能力。接下来,我们引入了FedKACE:1)一个自适应推理模型切换机制,使模型能够单向切换从本地模型到全局模型,以在个性化和泛化之间取得平衡;2)一个自适应梯度平衡重放方案,在类别重叠场景下协调新知识的学习和旧知识的保留;3)一个核谱边界缓冲区维护机制,保留高信息量和高边界影响的样本以优化跨轮次的知识保留。在多个场景中的实验和后悔分析证明了FedKACE的有效性。

英文摘要

Federated Continual Learning (FCL) leverages inter-client collaboration to better balance new knowledge acquisition and old knowledge retention on non-stationary data. However, existing FCL methods struggle to adapt to streaming scenarios where sequential and ephemerally accessible data chunks lack task identifiers and exhibit arbitrary class overlap, leading to confusion between old and new knowledge and an inability to sustain local inference on all encountered classes. To address this, we propose FedKACE with three components: 1) an adaptive mechanism that determines when to switch the inference model from the local to the global one to improve client-side inference performance; 2) a responsive gradient-balanced replay scheme that utilizes the ratio of the squared L2 gradient norms to balance client-specific knowledge between new acquisition and old retention; 3) a holistic buffer maintenance strategy that preserves highly informative and boundary-significant samples to enhance knowledge retention under class overlap.Experiments across multiple scenarios and theoretical analysis demonstrate the effectiveness of FedKACE.

URL PDF HTML 收藏
2510.21324 2026-07-17 cs.AI cs.MA 版本更新

CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretation

CXRAgent:用于胸部X光解读的由主任编排的多阶段推理

Jinhui Lou, Yan Yang, Zhou Yu, Zhenqi Fu, Weidong Han, Qingming Huang, Jun Yu

机构 * School of Computer Science, Hangzhou Dianzi University(杭州电子科技大学计算机科学学院) Department of Automation, Tsinghua University(清华大学自动化系) Department of Colorectal Medical Oncology, Zhejiang Cancer Hospital(浙江省肿瘤医院结直肠医学肿瘤科) School of Computer and Control Engineering, University of Chinese Academy of Sciences(中国科学院大学计算机与控制工程学院) School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)智能科学与工程学院)

AI总结 针对胸部X光解读中模型适应性差等问题,提出CXRAgent,通过中央主任协调多阶段,包括工具调用、诊断规划和协作决策,实验证明该方法性能出色,能提供视觉证据并适应不同临床任务。

Comments 10 pages, 4 figures, 7 Tables

详情
AI中文摘要

胸部X光在临床诊断中起关键作用,多种模型已用于自动解读。但这些模型难以适应新诊断任务和复杂推理场景。基于大语言模型的智能体模型成为胸部X光分析的有前景范式,不过现有智能体存在依赖单一诊断管道且缺乏评估工具可靠性机制的问题。为此提出CXRAgent,由中央主任协调多阶段:工具调用,用证据驱动验证器归一化和验证输出;诊断规划,根据任务要求和中间结果制定计划并组建专家团队;协作决策,整合专家见解和上下文记忆得出诊断结论。实验表明CXRAgent性能出色,能提供视觉证据且能很好地推广到不同复杂程度的临床任务。

英文摘要

Chest X-ray (CXR) plays a pivotal role in clinical diagnosis, and a variety of task-specific and foundation models have been developed for automatic CXR interpretation. However, these models often struggle to adapt to new diagnostic tasks and complex reasoning scenarios. Recently, LLM-based agent models have emerged as a promising paradigm for CXR analysis, enhancing model's capability through tool coordination, multi-step reasoning, and team collaboration, etc. However, existing agents often rely on a single diagnostic pipeline and lack mechanisms for assessing tools' reliability, limiting their adaptability and credibility. To this end, we propose CXRAgent, a director-orchestrated, multi-stage agent for CXR interpretation, where a central director coordinates the following stages: (1) Tool Invocation: The agent strategically orchestrates a set of CXR-analysis tools, with outputs normalized and verified by the Evidence-driven Validator (EDV), which grounds diagnostic outputs with visual evidence to support reliable downstream diagnosis; (2) Diagnostic Planning: Guided by task requirements and intermediate findings, the agent formulates a targeted diagnostic plan. It then assembles an expert team accordingly, defining member roles and coordinating their interactions to enable adaptive and collaborative reasoning; (3) Collaborative Decision-making: The agent integrates insights from the expert team with accumulated contextual memories, synthesizing them into an evidence-backed diagnostic conclusion. Experiments on various CXR interpretation tasks show that CXRAgent delivers strong performance, providing visual evidence and generalizes well to clinical tasks of different complexity. Code and data are valuable at this \href{https://github.com/laojiahuo2003/CXRAgent/}{link}.

URL PDF HTML 收藏
2406.10593 2026-07-17 cs.AI cs.DB cs.IR 版本更新

QDA-SQL: Questions Enhanced Dialogue Augmentation for Multi-Turn Text-to-SQL

QDA-SQL:用于多轮文本到SQL的问题增强对话扩充

Yinggang Sun, Ziming Guo, Haining Yu, Chuanyi Liu, Xiang Li, Bingxuan Wang, Xiangzhan Yu, Tiancheng Zhao

机构 * School of Cyberspace Science Harbin Institute of Technology Harbin, China(网络空间科学学院 哈尔滨工业大学 哈尔滨,中国) School of Computer Science(计算机科学学院) Technology Harbin University of Science(技术 哈尔滨理工大学) School of Cyberspace Science Harbin Institute of Technology, Shenzhen Shenzhen, China(网络空间科学学院 哈尔滨工业大学深圳 哈尔滨,中国)

AI总结 该研究针对多轮文本到SQL任务中微调模型面临的问题,提出QDA-SQL数据扩充方法,利用大语言模型生成多轮问答对,并引入验证和校正机制,提升了微调模型在SQL语句准确性及处理复杂问题上的能力。

Comments CAAI International Conference on Artificial Intelligence 2026 (CICAI 2026)

详情
AI中文摘要

针对特定领域任务微调大语言模型在文本到SQL任务中取得了巨大成功。然而,这些微调模型在多轮文本到SQL任务中常因问题模糊或无法回答而面临挑战。为解决此问题,我们提出了一种名为QDA-SQL的新型数据扩充方法,它使用大语言模型生成多种类型的多轮问答对。在QDA-SQL中,我们引入了一种结合验证和校正机制的方法来处理复杂的多轮文本到SQL任务。实验结果表明,QDA-SQL能使微调模型在SQL语句准确性上表现出更高性能,并增强其处理多轮文本到SQL任务中复杂、无法回答问题的能力。生成脚本和测试集已在该https URL发布。

英文摘要

Fine-tuning large language models (LLMs) for specific domain tasks has achieved great success in Text-to-SQL tasks. However, these fine-tuned models often face challenges with multi-turn Text-to-SQL tasks caused by ambiguous or unanswerable questions. It is desired to enhance LLMs to handle multiple types of questions in multi-turn Text-to-SQL tasks. To address this, we propose a novel data augmentation method, called QDA-SQL, which generates multiple types of multi-turn Q\&A pairs using LLMs. In QDA-SQL, we introduce a method incorporating validation and correction mechanisms to handle complex multi-turn Text-to-SQL tasks. Experimental results demonstrate that QDA-SQL enables fine-tuned models to exhibit higher performance on SQL statement accuracy and enhances their ability to handle complex, unanswerable questions in multi-turn Text-to-SQL tasks. The generation script and test set are released at https://github.com/mcxiaoxiao/QDA-SQL

URL PDF HTML 收藏
2607.13818 2026-07-16 cs.RO 新提交

Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learning

通过智能强化学习在机器人操作中学习鲁棒执行

Xiaopeng Zhang, Yueyang Weng, Qi Liu, Yongjin Mu, Yanjie Li

机构 * School of Inteligence Science and Engineering, the Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳)智能科学与工程学院) Faculty of Robot Science and Engineering, Northeastern University(东北大学机器人科学与工程学院)

AI总结 针对机器人操作面临的挑战,提出用两个互补指标评估执行质量,构建智能强化学习框架,通过高级决策恢复有效执行,在LIBERO基准测试中提升了执行成功率,增强了执行鲁棒性。

详情
AI中文摘要

机器人操作因不确定性、长期执行和复合误差面临根本挑战,易导致执行不稳定和任务失败。近期视觉语言动作(VLA)模型虽有强泛化能力,但缺乏评估执行稳定性及恢复偏离标称行为的机制。本文提出两个互补指标评估运行时执行质量,以及一个智能强化学习框架,通过高级决策恢复有效执行。该框架基于执行历史推理并选择执行模式调节执行过程,执行退化时触发恢复机制使任务继续。在LIBERO基准测试中评估,标准设置下成功率提高达13.7%,干扰设置下提高达39.2%,显著增强了执行鲁棒性。

英文摘要

Robotic manipulation poses fundamental challenges due to uncertainty, long-horizon execution, and compounding errors, which can easily destabilize execution and lead to task failure. Although recent vision-language-action (VLA) models exhibit strong generalization, they typically lack explicit mechanisms to assess execution stability and to recover when execution deviates from its nominal behavior. In this paper, we propose: (1) two complementary metrics to assess execution quality at runtime, and (2) an agentic reinforcement learning framework that learns to restore effective execution through high-level decision-making rather than directly learning low-level actions. In this framework, an agentic policy reasons over recent execution history and selects among a small set of execution modes to regulate the execution process. Under execution degradation, it triggers appropriate recovery mechanisms to restore the robot to previously visited nominal states, enabling the task to continue. We evaluate the proposed method on the LIBERO benchmark, achieving up to a 13.7% improvement in success rate under standard settings and up to a 39.2% improvement under disturbance settings, demonstrating substantially enhanced execution robustness.

URL PDF HTML 收藏
2607.13802 2026-07-16 cs.CV 新提交

RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics

RainDancer:基于面向降雨的脉冲动力学的RGB-事件视频去雨

Kui Jiang, Runzhe Li, Zhaocheng Yu, Guanglu Sun, Junjun Jiang, Xianming Liu

机构 * School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院) School of Computer Science and Technology, Harbin University of Science and Technology(哈尔滨理工大学计算机科学与技术学院)

AI总结 研究针对视频去雨问题,提出RainDancer框架,基于分解-交互范式,在RGB和事件分支分别处理,分离雨和背景成分,进行组件级融合,并引入事件域监督,实验表明该方法在多方面性能优越。

详情
AI中文摘要

视频去雨旨在从雨天视频中恢复干净的视觉内容,以便在恶劣天气下进行可靠感知。现有方法主要依赖RGB序列和时间冗余,但仅RGB恢复在动态雨景中仍不明确。事件相机提供具有高时间分辨率的互补运动敏感线索,但事件流也包含传感器噪声和背景触发响应。为解决此问题,我们提出RainDancer,一种基于分解-交互范式的渐进式RGB-事件视频去雨框架。核心思想是在跨模态交互之前在每个模态内分离雨和背景成分。在RGB分支中,帧特征被逐步分解为雨和背景表示。在事件分支中,一个面向降雨的脉冲神经网络模块捕获与雨运动相关的稀疏和突发事件动态。然后在语义对齐的表示之间进行组件级融合以保持结构和抑制降雨。我们还引入事件域监督来规范稀疏事件重建、结构一致性和梯度方向。在合成和真实RGB-事件视频去雨数据集上的实验证明了其优越的定量性能、视觉质量和下游感知鲁棒性。

英文摘要

Video deraining aims to recover clean visual content from rainy videos for reliable perception under adverse weather. Existing methods mainly rely on RGB sequences and temporal redundancy, but RGB-only restoration remains ambiguous in dynamic rainy scenes, where rain streaks, textures, boundaries, motion, and occlusions may share similar visual patterns. Event cameras provide complementary motion-sensitive cues with high temporal resolution, but event streams also contain sensor noise and background-triggered responses, so direct RGB-Event fusion may introduce cross-modal interference. To address this issue, we propose RainDancer, a progressive RGB-Event video deraining framework based on a decompose-before-interact paradigm. The core idea is to separate rain and background components within each modality before cross-modal interaction. In the RGB branch, frame features are progressively decomposed into rain and background representations. In the event branch, a rain-oriented spiking neural network module captures sparse and bursty event dynamics associated with rain motion. Component-level fusion is then performed between semantically aligned representations for structure preservation and rain suppression. We further introduce event-domain supervision to regularize sparse event reconstruction, structural consistency, and gradient orientation. Experiments on synthetic and real RGB-Event video deraining datasets demonstrate superior quantitative performance, visual quality, and downstream perception robustness. Code is available at https://github.com/AE86-plus/RainDancer.

URL PDF HTML 收藏
2607.13395 2026-07-16 cs.LG 新提交

Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

自我提升往往是突然的:大规模模型的顿悟式微调

Jing-Xiao Liao, Tianwei Zhang, Yu-Hao Jiang, Feifei Zhang, Hang-Cheng Dong, Feng-Lei Fan

机构 * School of Computer Science and Artificial Intelligence, Guangdong University of Education(广东第二师范学院计算机科学与人工智能学院) School of Instrumentation Science and Engineering, Harbin Institute of Technology(哈尔滨工业大学仪器科学与工程学院)

AI总结 研究大规模模型自主提升,提出顿悟式无训练后微调范式,通过修改关键模块捷径且不更新权重,针对大语言和视觉语言模型有不同实例化,实验证明有效解锁预训练网络潜力,提升性能。

详情
AI中文摘要

在大规模基础模型时代,追求自主自我提升的模型引发了越来越多的关注。从人类大脑中“顿悟”或“灵光一闪”的概念中获得灵感,我们假设大型模型表现出类似的顿悟现象——一种突然提升能力的潜在能力。然后,我们提出了顿悟(Enlightenment),一种用于大规模模型的新型无训练后微调范式。我们的方法在不更新权重的情况下修改关键模块/层的捷径,而现有的无训练方法主要操纵注意力权重。我们引入了两种特定于架构的实例化:i)对于大语言模型,我们提出了注意力头混合捷径,通过将初始注意力头的输出链接到所有其他目标头,并由自适应缩放因子初始化策略进行调制,来重新校准注意力权重。ii)对于视觉语言模型,我们在解码器层的残差连接中应用轻量级标量调制因子,调节信息流。大量实验表明,顿悟有效地解锁了预训练网络的潜在潜力,在各种基准和模型上产生了显著的性能提升。

英文摘要

The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models. Drawing inspiration from the concept of "enlightenment" or "aha moment" in human brain, we hypothesize that large models exhibit an analogous enlightenment phenomenon-a latent capacity for sudden capability boost. Then, we propose Enlightenment, a novel training-free post-tuning paradigm for large-scale models. Our approach modifies shortcuts for key modules/layers without weight updates, while existing training-free ones predominantly manipulate attention weights. We introduce two architecture-specific instantiations: i) For large language models, we propose attention head-mixing shortcuts that recalibrate attention weights by linking the initial attention head's output to all other target heads, modulated by an adaptive scaling factor initialization strategy. ii) For vision-language models, we apply a lightweight scalar-modulated factor to residual connections in the decoder layers, regulating information flow. Extensive experiments show that Enlightenment efficiently unlocks the latent potential of pre-trained networks, yielding remarkable performance improvements across diverse benchmarks and models.

URL PDF HTML 收藏
2607.12625 2026-07-16 cs.CL cs.CV 版本更新

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

KnowAct-GUIClaw:深度理解、完美执行,具有自我进化记忆和技能的个人GUI助手

Yunxin Li, Jinchao Li, Shibo Su, Zhenran Xu, Chenrui Zhao, Tongshu Bian, Xiaoman Liang, Meishan Zhang, Baotian Hu, Min Zhang

机构 * Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))

AI总结 针对OpenClaw的不足,提出深度理解、完美执行范式,介绍KnowAct-GUIClaw框架,通过多系统实验验证其在效率、准确性和跨平台适应性方面的优势,且知识记忆和执行技能可跨基础模型迁移。

Comments 29 pages, 9 figures

详情
AI中文摘要

OpenClaw作为复杂任务自动化的领先代理框架,存在跨平台GUI交互支持不足和自我进化机制不完善的问题,限制了其在不同设备生态系统中的应用及性能提升。本文提出个人助手的深度理解、完美执行范式,介绍了KnowAct-GUIClaw这一新颖框架,它能解决OpenClaw的GUI操作缺陷,突破跨平台和递归自我改进约束。通过在多系统上的广泛实验表明,KnowAct-GUIClaw具有卓越的效率、准确性和跨平台适应性,知识记忆和执行技能可跨多种基础模型迁移。

英文摘要

OpenClaw has emerged as a leading agent framework for complex task automation, yet it faces insufficient cross-platform GUI interaction support and a well-built self-evolution mechanism. These flaws limit its adaptation to diverse device ecosystems and prevent performance improvements through continuous learning from execution experience. To resolve these issues, we propose the Know Deeply, Act Perfectly paradigm for personal assistants, which holds that accumulated user interaction and task-running experience directly improve execution accuracy and efficiency, unifying cognitive comprehension and operational execution. Based on this paradigm, we introduce KnowAct-GUIClaw, a novel Know-Route-Act-Reflect framework designed to address OpenClaw's GUI manipulation deficits and break through its cross-platform and recursive self-improvement constraints. First, the host agent leverages accumulated interaction experience and task-relevant knowledge for long-horizon task decomposition and allocation (Know). Second, a pluggable GUI subagent with an experience-attributable memory system (Know) and self-evolving skill library (Act), enabling seamless cross-platform migration and fast-path integration. Especially, this framework continuously stores user profiles and feedback to improve the accuracy of task decomposition and tool calls. Extensive experiments across Android, iOS, HarmonyOS and Windows show that KnowAct-GUIClaw achieves superior efficiency, accuracy and cross-platform adaptability. Especially, the GUIClaw with open-source Kimi-2.6 models achieves the best performance (64.1%) on the long-horizon MobileWorld benchmark, beating all agentical frameworks and closed-source agentical models, e.g., Seed-2.0-Pro and GPT-5.5. Additionally, the knowledgeable memory and execution skills supported by our framework are transferable across diverse base models, improving by 8.5% with Kimi-2.6.

URL PDF HTML 收藏
2607.06540 2026-07-16 cs.CL 版本更新

Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

分层声学-语义建模:全双工口语语言模型的模态分离与语义连贯

Zhenyu Liu, Xuanyu Zhang, Yunxin Li, Qixun Teng, Shenyuan Jiang, Haolan Chen, Minjun Zhao, Fanbo Meng, Yu Xu, Yancheng He, Baotian Hu, Haizhou Li, Min Zhang

机构 * School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院) Center for Language, Intelligence and Machines, Shenzhen Loop Area Institute, Shenzhen(深圳环智中语言、智能与机器中心) School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)人工智能学院)

AI总结 研究全双工口语语言模型受模态干扰问题,提出Lychee-FD框架及分层参数分离策略,通过实验验证该方法能有效提升模型性能,在语音智能和交互流畅性上取得显著进步,且不影响推理效率。

Comments 22 pages, 9 figures, Fixed author

详情
AI中文摘要

开发无缝、高性能、原生智能的全双工口语语言模型(SLMs)一直是语音和自然语言处理社区的关键挑战和长期目标。尽管有显著进展,但近期努力受严重模态干扰根本限制,导致知识退化和语义完整性受损。本文通过对模型优化动态的详尽细粒度分析,揭示性能下降根源是声学和语义建模在共享深度参数空间时的固有梯度冲突。基于此,引入Lychee-FD框架,提出分层参数分离策略,在多个全双工基准测试上实验表明该方法显著提升了技术水平,在语音智能和全双工交互流畅性上有大幅改进且不影响推理效率。

英文摘要

Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity -- ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel. Extensive experiments on multiple full-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.

URL PDF HTML 收藏
2603.11811 2026-07-16 cs.RO cs.AI cs.CV 版本更新

RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Reset

RADAR: 通过语义规划和自主因果环境重置实现闭环机器人数据生成

Yongzhong Wang, Keyu Zhu, Yong Zhong, Liqiong Wang, Jinyu Yang, Feng Zheng

机构 * Southern University of Science and Technology(南方科技大学) Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)) Spatialtemporal AI(时空人工智能)

AI总结 RADAR通过语义规划和自主因果环境重置实现闭环机器人数据生成,具备高适应性和可扩展性,在仿真和现实部署中均表现出卓越的性能。

Comments 8 pages, 4 figures. Accepted to the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026). Project page: https://radar-iros.netlify.app/

详情
AI中文摘要

大规模物理交互数据的获取,是现代机器人学习的关键前提,但受到人机交互收集范式的高昂成本和可扩展性限制而严重瓶颈。为突破这一障碍,我们引入了Robust Autonomous Data Acquisition for Robotics(RADAR),一种完全自主、闭环的数据生成引擎,完全消除了数据收集过程中的人工干预。RADAR巧妙地将认知负荷分解为一个四模块流水线。以2-5个3D人类演示作为几何先验,首先由视觉-语言模型通过精确的语义对象定位和技能检索来 orchestrate 与场景相关的任务生成。接着,图神经网络策略将这些子任务转化为物理动作,通过上下文模仿学习。执行后,VLM 使用结构化的视觉问答管道进行自动化成功评估。最后,为打破手动重置的瓶颈,有限状态机协调自主环境重置和不对称数据路由机制。通过同时的正反向规划,严格遵循Last-In, First-Out因果序列,系统无缝地恢复无结构的工作空间并稳健地从执行失败中恢复。这种持续的脑-小脑协同将数据收集转化为一个自维持的过程。广泛的评估突显了RADAR的卓越适应性。在仿真中,我们的框架在复杂、长时间任务上实现了高达90%的成功率,轻松解决了传统基线几乎降至零性能的挑战。在现实部署中,系统通过少样本适应可靠地执行多样、接触丰富的技能(例如变形物体操作),无需领域特定的微调,提供了一种高度可扩展的机器人数据获取范式。

英文摘要

The acquisition of large-scale physical interaction data, a critical prerequisite for modern robot learning, is severely bottlenecked by the prohibitive cost and scalability limits of human-in-the-loop collection paradigms. To break this barrier, we introduce Robust Autonomous Data Acquisition for Robotics (RADAR), a fully autonomous, closed-loop data generation engine that completely removes human intervention from the collection cycle. RADAR elegantly divides the cognitive load into a four-module pipeline. Anchored by 2-5 3D human demonstrations as geometric priors, a Vision-Language Model first orchestrates scene-relevant task generation via precise semantic object grounding and skill retrieval. Next, a Graph Neural Network policy translates these subtasks into physical actions via in-context imitation learning. Following execution, the VLM performs automated success evaluation using a structured Visual Question Answering pipeline. Finally, to shatter the bottleneck of manual resets, a Finite State Machine orchestrates an autonomous environment reset and asymmetric data routing mechanism. Driven by simultaneous forward-reverse planning with a strict Last-In, First-Out causal sequence, the system seamlessly restores unstructured workspaces and robustly recovers from execution failures. This continuous brain-cerebellum synergy transforms data collection into a self-sustaining process. Extensive evaluations highlight RADAR's exceptional versatility. In simulation, our framework achieves up to 90% success rates on complex, long-horizon tasks, effortlessly solving challenges where traditional baselines plummet to near-zero performance. In real-world deployments, the system reliably executes diverse, contact-rich skills (e.g., deformable object manipulation) via few-shot adaptation without domain-specific fine-tuning, providing a highly scalable paradigm for robotic data acquisition.

URL PDF HTML 收藏
2607.12896 2026-07-15 cs.CV 新提交

UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation

UniMedSeg:用于多范式2D/3D医学图像分割的统一上下文学习

Yunzhou Li, Jiesi Hu, Yanwu Yang, Hanyang Peng, Chenfei Ye, Jianfeng Cao, Yixuan Yuan, Ting Ma

机构 * Harbin Institute of Technology at Shenzhen(哈尔滨工业大学(深圳)) Peng Cheng Laboratory(鹏城实验室) University Hospital Tübingen(图宾根大学医院) German Center for Mental Health(德国心理健康中心) Chinese University of Hong Kong(香港中文大学)

AI总结 研究针对医学图像分割基础模型存在的问题,提出以Transformer为中心的UniMedSeg框架,通过映射多种信息到共享序列空间联合学习异构医学监督,引入解耦分割注意力克服内存瓶颈,在多类分割任务中实现最优性能且无需特定任务微调。

详情
AI中文摘要

医学图像分割基础模型期望能在不同临床场景中通用,但现有通用方法因提示范式和空间维度而碎片化。视觉上下文学习、交互式分割和语言引导分割通常由特定范式模型处理,2D和3D图像也单独建模。这种隔离阻碍了异构注释和数据被单个可扩展模型联合吸收,限制了跨范式知识转移。为解决此瓶颈,我们提出UniMedSeg,一个以Transformer为中心的通用分割框架,将视觉示例、几何交互、语言指令和2D/3D图像映射到共享序列空间,通过统一上下文接口联合学习异构医学监督,无需特定提示或维度分支。为克服视觉上下文导致的长序列内存瓶颈,我们引入解耦分割注意力,将注意力复杂度降至线性,同时保持硬件友好计算和聚焦的上下文-目标交互。在从27个公共数据集策划的大型语料库上进行广泛训练和评估,UniMedSeg在视觉上下文、交互式和语言引导分割中实现了无特定任务微调的最优性能,证明了在不同保留任务上的强大泛化能力。代码和模型权重可在指定网址公开获取。

英文摘要

Medical image segmentation foundation models are expected to generalize across diverse clinical scenarios, yet existing universal methods remain fragmented by prompt paradigms and spatial dimensions. Visual in-context learning, interactive segmentation, and language-guided segmentation are typically handled by paradigm-specific models, while 2D and 3D images are also modeled separately. Such isolation prevents heterogeneous annotations and data from being jointly absorbed by a single scalable model and limits cross-paradigm knowledge transfer. To address this bottleneck, we propose UniMedSeg, a Transformer-centric universal segmentation framework that maps visual examples, geometric interactions, language instructions, and 2D/3D images into a shared sequence space, enabling heterogeneous medical supervision to be jointly learned through a unified in-context interface without prompt- or dimension-specific branches. To overcome the long-sequence memory bottleneck caused by visual contexts, we introduce Decoupled Split Attention, which reduces attention complexity to linear while preserving hardware-friendly computation and focused context-target interaction. Extensively trained and evaluated on a large corpus curated from 27 public datasets, UniMedSeg achieves state-of-the-art performance across visual in-context, interactive, and language-guided segmentation without task-specific fine-tuning, demonstrating strong generalization on diverse held-out tasks. The code and model weights are publicly available at https://github.com/Lii1228/UniMedSeg

URL PDF HTML 收藏