Large language models (LLMs) are trained and tested extensively on symbolic representations such as code and graphs, yet real-world user tasks are often specified in natural language. To what extent can LLMs generalize across these representations? Here, we approach this question by studying isomorphic tasks involving procedures represented in code, graphs, and natural language (e.g., scheduling steps in planning). We find that training LLMs with popular post-training methods on graphs or code data alone does not reliably generalize to corresponding natural language tasks, while training solely on natural language can lead to inefficient performance gains. To address this gap, we propose a two-stage reinforcement learning curriculum that first trains on symbolic, then natural language data. The curriculum substantially improves model performance across model families and tasks. Remarkably, a 1.5B Qwen model trained by our method can closely match zero-shot GPT-4o in naturalistic planning. Finally, our analysis suggests that successful cross-representation generalization can be interpreted as a form of generative analogy, which our curriculum effectively encourages. The dataset and code used in this paper can be found \href{https://github.com/fangru-lin/procedure_generalization_llm}{here}.
A Retrieval-Based Approach to Medical Procedure Matching in Romanian
National University of Science and Technology POLITEHNICA Bucharest（波兰技术大学科学与技术国家大学）；Dalle Molle Institute for Artificial Intelligence Research (IDSIA)（达勒莫利人工智能研究所）
HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish
Institute of Computer Science, Polish Academy of Sciences（波兰科学院计算机科学研究所）；AGH University of Science and Technology（AGH科技大学）
基于BERT的模型目前被用于解决几乎所有的自然语言处理（NLP）任务，并且大多数情况下都能获得最先进的结果。因此，NLP社区对理解这些模型进行了广泛的研究，但最重要的是设计有效和高效的培训程序。已经进行了几项消融研究，调查如何训练BERT类模型，但其中绝大多数只涉及英语。为英语设计的培训程序不一定是通用的，也不一定适用于其他类型上特别不同的语言。因此，本文提出了第一个消融研究集中在波兰语，这与孤立的英语语言，是一个融合的语言。我们设计并全面评估了将知识从多语言模型转移到单语言BERT模型的预训练过程。除了多语言模型初始化之外，还探讨了可能影响预训练的其他因素，即训练目标、语料库大小、BPE-Dropout和预训练长度。基于所提出的过程，波兰BERT为基础的语言模型- HerBERT -训练。该模型在多个下游任务上实现了最先进的结果。
Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA
NLP；CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学CT实验室）；Department of Electrical and Computer Engineering, University of Macau（电气与计算机工程系，澳门大学）；Centre for Cognitive and Brain Sciences, University of Macau（认知与脑科学中心，澳门大学）
