LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
National University of Singapore(新加坡国立大学);Nanjing University(南京大学);University of Toronto(多伦多大学);Mila, Québec AI Institute(魁北克人工智能研究所(Mila));Université de Montréal(蒙特利尔大学);Tencent Inc(腾讯公司);Baidu Inc(百度公司);Nanyang Technological University(南洋理工大学)
Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. Meanwhile, extending the context window in LLMs through post-pretraining is highly resource-intensive. To address this, we introduce LongRecipe, an efficient training strategy for extending the context window of LLMs, including impactful token analysis, position index transformation, and training optimization strategies. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training. Furthermore, LongRecipe also preserves the original LLM's capabilities in general tasks. Ultimately, we can extend the effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with 80G memory. Our code is released at https://github.com/zhiyuanhubj/LongRecipe.
RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?
University of California, Berkeley(加州大学伯克利分校);University of Wisconsin, Madison(威斯康星大学麦迪逊分校);University of Washington(华盛顿大学);Ai2 Main dataset contributor(Ai2 主数据贡献者)
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Qwen Large Model Application Team, Alibaba Institute of Automation, Chinese Academy of Sciences(文勤大模型应用团队,阿里巴巴自动化研究所,中国科学院);Qwen Large Model Application Team, Alibaba(文勤大模型应用团队,阿里巴巴)
The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
Northeastern University(东北大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Northwestern Polytechnical University(西北工业大学);Beijing Institute of Technology(北京理工大学);The University of Tokyo(东京大学);Singapore Management University(新加坡管理学院)
Spec-o3: A Tool-Augmented Vision-Language Agent for Rare Celestial Object Candidate Vetting via Automated Spectral Inspection
SKL-MAIS, Institute of Automation, CAS(SKL-MAIS,自动化研究所,中国科学院);School of Advanced Interdisciplinary Sciences, UCAS(交叉科学学院,中国科学院大学);National Astronomical Observatories, CAS(国家天文台,中国科学院);School of Artificial Intelligence, UCAS(人工智能学院,中国科学院大学)