Intrinsic Entropy of Context Length Scaling in LLMs
Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；Carnegie Mellon University（卡内基梅隆大学）；CPHOS；University of Washington（华盛顿大学）
Jingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao, Jeng-Neng Hwang, Lei Li
Long Context Language Models have drawn great attention in the past few years. There has been work discussing the impact of long context on Language Model performance: some find that long irrelevant context could harm performance, while some experimentally summarize loss reduction by relevant long context as Scaling Laws. This calls for a more thorough understanding of how long context impacts Language Modeling. In this work, we (1) propose to use `Intrinsic Entropy' for explaining the impact of context length on language modeling; and (2) conduct experiments on natural language and synthetic data, validating our proposed theoretical assumptions and deductions. Our theoretical framework can provide practical insights such as establishing that training dataset size dictates an optimal context length and bounds context length scaling for certain cases. We hope our work may inspire new long context Language Models, as well as future work studying the physics of Language Models.
Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
Princeton University（普林斯顿大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Toronto（多伦多大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Stanford University（斯坦福大学）；University of Oxford（牛津大学）
When Should LLMs Reason? Adaptive Inference Routing via Entropy Dynamics
Samsung Research（三星研究院）；State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，北京理工大学）
Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?
RISE Research Institutes of Sweden, Department of Computer Science（瑞典RISE研究机构，计算机科学系）；Uppsala University, Department of Linguistics and Philology（乌普萨拉大学，语言学与哲学系）；KTH Royal Institute of Technology, Division of Software and Computer Systems（皇家理工学院，软件与计算机系统系）；University of Edinburgh, School of Informatics（爱丁堡大学，信息学院）
Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMs
State Key Laboratory of CCSE,School of Computer Science and Engineering,Beihang University（信息与通信系统国家重点实验室，计算机科学与工程学院，北京航空航天大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Institute of Technology（北京理工大学）
Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China（天津大学计算机科学与技术学院 TJUNLP 实验室，中国）；Independent Researcher（独立研究者）；College of Computer and Information Engineering, Tianjin Normal University, China（天津师范大学计算机与信息工程学院，中国）
Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures
MOE Key Lab of Computational Linguistics, Peking University（计算语言学MOE实验室，北京大学）；Beijing Academy of Artificial Intelligence, Beijing, China（北京人工智能研究院，北京，中国）；Nanjing University of Science and Technology（南京理工大学）；Purdue University（普渡大学）
Department of School of Engineering, Shenzhen MSU-BIT University, Shenzhen, China, 518000（深圳MSU-BIT大学工程学院部门）；MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU-BIT University, Shenzhen, China, 518000（应用数学联合研究中心）
Entropy Sentinel: Probing Entropy Traces for LLM Monitoring
Departamento de Matematica, FCEyN Universidad de Buenos Aires（数学系，布宜诺斯艾利斯大学）；ELIAS Lab, Departamento de Ingeniería Universidad de San Andres（ELIAS实验室，圣安德烈斯大学）
Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMs
International Institute of Information Technology, Hyderabad（国际信息研究所，海得拉巴）；Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi（Mohamed bin Zayed人工智能大学，阿布扎比）
Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models
Department of Electrical and Electronic Engineering（电子与电气工程系）；The Hong Kong Polytechnic University（香港理工大学）；School of Electronics and Information（电子与信息学院）；Northwestern Polytechnical University（西北工业大学）
Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lens
School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China（计算机科学与工程学院、计算机网络与信息集成重点实验室、教育部、东南大学，中国）；Department of Neurosurgery, Shanghai Tenth People’s Hospital, School of Clinical Medicine of Nanjing Medical University, China（神经外科科、上海第十人民医院、南京医科大学临床医学学院，中国）
