Despite the advances in Large Language Models (LLMs), exemplified by models like GPT-4 and Claude, smaller-scale LLMs such as Llama and Mistral often struggle with generating in-depth and coherent dialogues. This paper presents a novel two-step Coarse-to-Fine Actor model to address the inherent limitations in conversational and analytical capabilities of small-sized LLMs. Our approach begins with the Policy-based Coarse Actor, employing a technique we term "Continuous Maximization". The Coarse Actor establishes an enhanced, knowledge-rich pool adept at aligning with human preference styles in analysis and reasoning. Through the RLHF process, it employs Continuous Maximization, a strategy that dynamically and adaptively extends the output length limit, enabling the generation of more detailed and analytical content. Subsequently, the Fine Actor refines this analytical content, addressing the generation of excessively redundant information from the Coarse Actor. We introduce a "Knowledge Residue Merger" approach, refining the content from the Coarse Actor and merging it with an existing Instruction model to improve quality, correctness, and reduce redundancies. We applied our methodology to the popular Mistral model, creating Mistral-C2F, which has demonstrated exceptional performance across 11 general language tasks and the MT-Bench Dialogue task, outperforming similar-scale models and even larger models with 13B and 30B parameters. Our model has significantly improved conversational and analytical reasoning abilities.
Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance
Indian Institute of Technology Kharagpur（印度理工学院克勒格布尔分校）；National Institute of Standards and Technology（美国国家标准与技术研究院）；INSAIT（保加利亚索菲亚大学INSAIT研究院）；Sofia University “St. Kliment Ohridski”（圣基里曼·奥赫里德斯基索菲亚大学）
PatientDx: Merging Large Language Models for Protecting Data-Privacy in Healthcare
Université de Toulouse（图卢兹大学）；IRIT UMR 5505；Centre Hospitalier Universitaire de Toulouse（图卢兹大学医院中心）；INSERM（法国国家健康与医学研究院）；Faculté de Médecine Université de Toulouse（图卢兹大学医学院）
Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Lab of AI Safety（人工智能安全国家重点实验室）；University of Chinese Academy of Sciences, CAS（中国科学院大学）；Zhejiang University（浙江大学）
