Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution. StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts and converting token-level log-probability gaps into sign-preserving advantage shaping with a normalized per-step credit budget before the GRPO update. Across ALFWorld and Search-QA with Qwen3-1.7B and Qwen2.5-3B-Instruct, StepOPSD attains best or second-best results on subsets most sensitive to local causal errors, including first-place performance on ALFWorld Heat (79.1%), PickTwo (95.0%), Search-QA TriviaQA (61.6%), and tied-best performance on HotpotQA (40.4%). The results further reveal a consistent two-knob law: smaller α_clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength λ_mix remains task-dependent. These findings suggest that step-aware distillation is most useful when trajectory-level rewards are weakly aligned with the local action that determines downstream success.
Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算高精尖创新中心);Tsinghua University(清华大学);Beihang University(北京航空航天大学)
Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Soochow University(苏州大学);University of Technology Sydney(悉尼科技大学);The Hong Kong University of Science and Technology(香港科技大学)
针对成对偏好蒸馏忽视回复差异程度的问题,本文提出 PAD 框架,将师生内在偏好建模为分布并进行对齐,在多个基准上显著提升 SLM 的人类偏好对齐效果。
Active Multimodal Distillation for Few-shot Action Recognition
College of Computer and Information Engineering, Tianjin Normal University(天津师范大学计算机与信息工程学院);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));College of Intelligence and Computing, Tianjin University(天津大学智能科学与计算学院);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江省区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou(杭州高新技术区(滨江)区块链与数据安全研究院)
BPL: Bias-adaptive Preference Distillation Learning for Recommender System
Department of Computer Science and Engineering, Korea University(计算机科学与工程系,韩国大学);Microsoft Research Asia(微软亚洲研究院);Department of Aritifial Intelligence, Yonsei University(人工智能系,延世大学);Department of Computer Science and Engineering, POSTECH(计算机科学与工程系,POSTECH);William & Mary(威廉与玛丽学院)
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education(大数据与人工智能交通运输联合实验室,(北京交通大学)教育部);School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China(计算机科学与技术学院,北京交通大学,北京,中国);Tencent Inc, China(腾讯公司,中国)
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);University of Chicago(芝加哥大学);The Chinese University of Hong Kong(香港中文大学);East China Normal University(华东师范大学)
Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS);Meituan(美团)