Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous agents typically rely on complex prompting or extensive fine-tuning, which often fails to introduce new capabilities while preserving strong generalizability. We introduce Hephaestus-Forge, the first large-scale pre-training corpus designed to enhance the fundamental capabilities of LLM agents in API function calling, intrinsic reasoning and planning, and adapting to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data encompassing 76,537 APIs, including both tool documentation to introduce knowledge of API functions and function calling trajectories to strengthen intrinsic reasoning. To explore effective training protocols, we investigate scaling laws to identify the optimal recipe in data mixing ratios. By continual pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale open-source LLMs and rivals commercial LLMs on three agent benchmarks, demonstrating the effectiveness of our pre-training corpus in enhancing fundamental agentic capabilities and generalization of LLMs to new tasks or environments.
Stability-Plasticity Balance via Singular-Vector Selection in LLM Continual Learning
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(未来区块链与隐私计算北京高精尖创新中心);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
East China Normal University(东华大学);Shanghai Innovation Institute(上海创新研究院);Shanghai AI Laboratory(上海人工智能实验室);Xiamen University(厦门大学);Tencent YouTu Lab(腾讯YouTu实验室)
Learn it or Leave it: Module Composition and Pruning for Continual Learning
Bosch Center for Artificial Intelligence(博世人工智能中心);LMU Munich(慕尼黑大学);Hochschule der Medien(慕尼黑应用科技大学);Karlsruhe University of Applied Sciences(卡尔斯鲁厄应用科技大学)
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory(协调科学实验室);Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Mathematics(数学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Knowledge Fusion of Large Language Models Via Modular SkillPacks
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Hong Kong Polytechnic University(香港理工大学);Nanyang Technological University(南洋理工大学);Huazhong University of Science and Technology(华中科技大学)
Continually Learn to Map Visual Concepts to Large Language Models in Resource-constrained Environments
Bordeaux INP - ENSEIRB-MATMECA(波尔多 INP - ENSEIRB-MATMECA);CAMaCS, University of Warwick(Warwick 大学 CAMaCS 实验室);Institute of Computer Science Polish Academy of Sciences(波兰科学院计算机科学研究所);University of Pisa(比萨大学)
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology(大连理工大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学)
Multi-Level Safety Continual Projection for Fine-Tuned Large Language Models without Retraining
Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences(脑启发认知人工智能实验室,中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Long-term AI(长期人工智能)
From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);New Jersey Institute of Technology(新泽西理工学院)
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Division of Statistics and Machine Learning (STIMA), Department of Computer and Information Science (IDA), Linköping University(统计与机器学习系(STIMA)、计算机与信息科学系(IDA)、利厄普堡大学)
K-DeCore: Facilitating Knowledge Transfer in Continual Structured Knowledge Reasoning via Knowledge Decoupling
School of Computer Science and Engineering, Southeast University, China(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及 interdisciplinary 应用关键实验室(东南大学));China Mobile Research Institute(中国移动研究院);Department of Data Science & AI, Monash University, Australia(莫纳什大学数据科学与人工智能系)
SafeCoEvo: Co-Evolving Safety Harnesses and Guards for LLM Agents at Test-Time
East China Normal University(华东师范大学);Shanghai Innovation Institute(上海创新研究院);Xiamen University(厦门大学);Harbin Institute of Technology(哈尔滨工业大学);University College London(伦敦大学学院);Huawei Noah’s Ark Lab, UK(华为诺亚方舟实验室(英国));MemoraX AI
Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs
The PII shown in the figure is synthetic and not real(合成的PII);International Digital Economy Academy(国际数字经济学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);DataArc Tech Ltd(DataArc科技有限公司)