The deployment of Large Language Models (LLMs) in long-context scenarios is hindered by computational inefficiency and significant information redundancy. Although recent advancements have widely adopted context compression to address these challenges, existing research only focus on model-side improvements, the impact of the data distribution itself on context compression remains largely unexplored. To bridge this gap, we are the first to adopt a data-centric perspective to systematically investigate how data distribution impacts compression quality, including two dimensions: input data and intrinsic data (i.e., the model's internal pretrained knowledge). We evaluate the semantic integrity of compressed representations using an autoencoder-based framework to systematically investigate it. Our experimental results reveal that: (1) encoder-measured input entropy negatively correlates with compression quality, while decoder-measured entropy shows no significant relationship under a frozen-decoder setting; and (2) the gap between intrinsic data of the encoder and decoder significantly diminishes compression gains, which is hard to mitigate. Based on these findings, we further present practical guidelines to optimize compression gains.
Shifting AI Efficiency From Model-Centric to Data-Centric Compression
EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）；Sichuan University（四川大学）；University of Electronic Science & Technology of China（电子科技大学）；Shanghai AI Laboratory（上海人工智能实验室）；Sun Yat-sen University（中山大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
A Survey on Efficient Large Language Model Training: From Data-centric Perspectives
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University（信息处理国家重点实验室，计算机学院，PKU-Anker LLM实验室，北京大学）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Washington（华盛顿大学）；Georgia Institute of Technology（佐治亚理工学院）；Nanyang Technological University（南洋理工大学）；HKUST（香港科技大学）；University of International Business and Economics（国际商务经济大学）
Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind
X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China（X-LANCE实验室，计算机科学与工程系，MoE人工智能关键实验室，上海交通大学人工智能研究院，上海，中国）；Suzhou Laboratory, Suzhou, China（苏州实验室，苏州，中国）
A Survey of Multimodal Large Language Model from A Data-centric Perspective
Hong Kong University of Science and Technology（香港科技大学）；Apple（苹果公司）；Peking University（北京大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Science and Technology of China（中国科学技术大学）
Revealing the Technology Development of Natural Language Processing: A Scientific Entity-Centric Perspective
Department of Information Management, Nanjing University of Science and Technology（南京理工大学信息管理学院）；Department of Management Science and Engineering, School of Management, Anhui University（安徽大学管理科学与工程学院）
QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory
State Key Laboratory of AI Safety, ICT, CAS（人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Assessing Knowledge Editing in Language Models via Relation Perspective
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, CAS（中国科学院自动化研究所）；Kunming University of Science and Technology（昆明理工大学）；Beijing Institute of Technology（北京理工大学）
UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective
The University of Hong Kong（香港大学）；Nanjing University（南京大学）；University College London（伦敦大学学院）；The Ohio State University（俄亥俄州立大学）；Microsoft Research Asia（微软亚洲研究院）；The Chinese University of Hong Kong（香港中文大学）；SIAT, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion
Computation and Artificial Intelligence Innovative College, Fudan University（复旦大学计算与人工智能创新学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai University of International Business and Economics（上海国际商务经济学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Rethinking Soft Compression in Retrieval-Augmented Generation: A Query-Conditioned Selector Perspective
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence（上海数据科学 key 实验室，计算机科学与人工智能学院）；Fudan University（复旦大学）；Shanghai（上海）；China（中国）
Text Distance from Nested and Hierarchical Repetitions: A Compression-Based Perspective
Department of Systems Science, Faculty of Arts and Sciences, Beijing Normal University（北京师范大学文理学院系统科学系）；International Academic Center of Complex Systems, Beijing Normal University（北京师范大学复杂系统国际学术中心）；School of Systems Science, Beijing Normal University（北京师范大学系统科学学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件学院）
