CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China(东北大学计算机科学与工程学院);Tsinghua University(清华大学);Future Living Lab of Alibaba(阿里巴巴未来生活实验室)
The quadratic complexity and indefinitely growing key-value (KV) cache of standard Transformers pose a major barrier to long-context processing. To overcome this, we introduce the Collaborative Memory Transformer (CoMeT), a novel architecture that enables LLMs to handle arbitrarily long sequences with constant memory usage and linear time complexity. Designed as an efficient, plug-in module, CoMeT can be integrated into pre-trained models with only minimal fine-tuning. It operates on sequential data chunks, using a dual-memory system to manage context: a temporary memory on a FIFO queue for recent events, and a global memory with a gated update rule for long-range dependencies. These memories then act as a dynamic soft prompt for the next chunk. To enable efficient fine-tuning on extremely long contexts, we introduce a novel layer-level pipeline parallelism strategy. The effectiveness of our approach is remarkable: a model equipped with CoMeT and fine-tuned on 32k contexts can accurately retrieve a passkey from any position within a 1M token sequence. On the SCROLLS benchmark, CoMeT surpasses other efficient methods and achieves performance comparable to a full-attention baseline on summarization tasks. Its practical effectiveness is further validated on real-world agent and user behavior QA tasks. The code is available at: https://github.com/LivingFutureLab/Comet
Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models
UCL Knowledge Lab, Institute of Education, University College London, UK(伦敦大学学院教育研究所知识实验室);AI Centre, Dept. of Computer Science, University College London, UK(伦敦大学学院人工智能中心)
COMET: Convolutional Dimension Interaction for Collaborative Filtering
Nanyang Technological University(南洋理工大学);Stony Brook University(石溪大学);The Institute of Cancer Research(英国癌症研究院);University of Florida(佛罗里达大学);Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局(A*STAR)新加坡制造技术研究院(SIMTech))
Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks
Pillar of Information Systems Technology and Design, Singapore University of Technology and Design(信息系统技术与设计学院,新加坡南洋理工大学);School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast(电子、电气与计算机科学学院,女王大学贝尔法斯特分校);Department of Electrical and Computer Engineering, Sungkyunkwan University(电气与计算机工程系,成均馆大学);KU 6G Research Center, Department of Computer and Information Engineering, Khalifa University(6G研究中心,计算机与信息工程系,哈利法大学);CentraleSupelec, University of Paris-Saclay(中央超导研究所,巴黎萨克雷大学);Department of Electrical and Computer Engineering, Princeton University(电气与计算机工程系,普林斯顿大学)
The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP
University of California, Los Angeles(加州大学洛杉矶分校);Georgia Institute of Technology(佐治亚理工学院);University of Wisconsin - Madison(威斯康星大学麦迪逊分校);University of Cape Coast(科克伍德大学);Carleton University(卡尔顿大学);Stetson University(史蒂文森大学);Northwestern University in Qatar(卡塔尔西北大学);Cornell University(康奈尔大学);Soka University of America(美国立命大学);Columbia University(哥伦比亚大学)
Center for Artificial Intelligence and Technology(人工智能与技术中心);Faculty of Information Science and Technology(信息科学与技术学院);The National University of Malaysia(马来西亚国立大学)
CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks
State Key Laboratory of Networking and Switching Technology(网络与交换技术国家重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Singapore University of Technology and Design(新加坡科技设计大学);School of Electronics, Electrical Engineering and Computer Science(电子、电气与计算机科学学院);Queen’s University Belfast(贝尔法斯特女王大学);Dept. of Electrical Engineering & Computer Science(电气与计算机科学系);Lassonde School of Engineering(拉索nde 工程学院);York University(约克大学);College of Computing and Data Science(计算与数据科学学院);Nanyang Technological University(南洋理工大学)
Learning to Iteratively Solve Routing Problems with Dual-Aspect Collaborative Transformer
National University of Singapore(新加坡国立大学);University of Electronic Science and Technology of China(电子科技大学);Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);The Hong Kong University of Science and Technology(香港科技大学);Singapore Institute of Manufacturing Technology, A*STAR(新加坡科技研究局新加坡制造技术研究院);Shandong University(山东大学)
Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);IoT Thrust and Research Center for Digital World with Intelligent Things, HKUST (Guangzhou)(香港科技大学(广州)智能物联数字世界研究中心及物联网学科);Data Science and Analytics Thrust, HKUST (Guangzhou)(香港科技大学(广州)数据科学与分析学科);Department of Computer Science and Engineering, HKUST(香港科技大学计算机科学与工程系)
TransGNN: Harnessing the Collaborative Power of Transformers and Graph Neural Networks for Recommender Systems
Hong Kong University of Science and Technology(香港科技大学);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Interdisciplinary Institute for Medical Engineering, Fuzhou University(福州大学医学工程跨学科研究院);Microsoft Research Asia(微软亚洲研究院);Central South University(中南大学);Jinan University(暨南大学)
Collaborative Temporal Feature Generation via Critic-Free Reinforcement Learning for Cross-User Sensor-Based Activity Recognition
School of Artificial Intelligence (School of Future Technology), Nanjing University of Information Science and Technology(信息科学与技术大学人工智能学院);School of Cyberspace Security, Nanjing University of Information Science and Technology(信息科学与技术大学网络空间安全学院);Jinling Hospital, Affiliated Hospital of Medical School, Nanjing University(南京大学附属医院金陵医院);Department of Electrical, Computer, and Software Engineering, The University of Auckland(奥克兰大学电气、计算机和软件工程系)