Context distillation compresses contextual information into model parameters, yet existing methods often ignore how multiple distilled latent memories should be stored, retrieved, and safely activated in non-oracle settings. We formulate context distillation as a latent memory management problem. We distill each context into an independent LoRA adapter, forming a modular memory bank that enables explicit memory selection. Given a query, our framework retrieves candidate memories, routes the query to the most suitable adapter, and uses a Self-Gating mechanism to decide whether latent memory should be activated. To improve efficiency, we further introduce cache sharing to reduce management overhead during inference. Experiments show that our method substantially outperforms baselines with retrieval, while Self-Gating improves robustness by deactivate unnecessary latent memories.
TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation
University of Science and Technology of China(中国科学技术大学);Beijing University of Posts and Telecommunications(北京邮电大学);Rutgers Business School(罗格斯商学院);Sun Yat-sen University(中山大学)
scDD: Latent Codes Based scRNA-seq Dataset Distillation with Foundation Model Knowledge
National Institute of Health Data Science, Peking University(北京大学健康数据科学国家研究院);Institute of Medical Technology, Peking University Health Science Center(北京大学医学部医学技术研究院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);AI Research Institute, China Mobile Communications Corporation(中国移动通信集团人工智能研究院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究院)
Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits
Tsinghua University(清华大学);University of California, Los Angeles(加州大学洛杉矶分校);Peking University(北京大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)
Teacher-free Latent Self-distillation and Class-separable Representations for Lightweight IoT Attack Detection
School of Electrical and Data Engineering, the University of Technology Sydney(电气与数据工程学院,悉尼大学技术学院);Computer Science Department, Institute of Information and Communication Technology, Le Quy Don Technical University(计算机科学系,信息与通信技术研究所,吕公独技术大学);University of Engineering and Technology, Vietnam National University(工程与技术大学,越南国家大学)
LastOPD: Taming Collapse in Latent On-Policy Distillation
University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Visa Research(Visa研究院);Case Western Reserve University(凯斯西储大学);University of Florida(佛罗里达大学);The Ohio State University(俄亥俄州立大学)
Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);The University of Hong Kong(香港大学);The City University of New York(纽约城市大学);Tsinghua University(清华大学)
Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
University of British Columbia(不列颠哥伦比亚大学);University of Helsinki(赫尔辛基大学);ELLIS Institute Finland(芬兰ELLIS研究所);University of Turku(图尔库大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
DISC: Latent Diffusion Models with Self-Distillation from Separated Conditions for Prostate Cancer Grading
Scientific Computing and Imaging Institute, University of Utah(犹他大学科学计算与成像研究所);The Catholic University of Korea College of Medicine(韩国天主教大学医学院);Case Western Reserve University(凯斯西储大学);Huntsman Cancer Institute, University of Utah Health(犹他大学健康亨茨曼癌症研究所);University of Utah(犹他大学)