ICE-GRT：基于生成式强化Transformer的指令上下文增强
ICE-GRT: Instruction Context Enhancement by Generative Reinforcement based Transformers
AI总结：
本文提出ICE-GRT模型，通过基于PPO的RLHF技术，在保持通用能力的同时提升特定领域任务的深度和分析能力，实现SOTA性能。
摘要：
ChatGPT和LLaMA等大型语言模型（LLMs）的出现面临特定领域任务的局限性，这些模型在专业领域通常缺乏深度和准确性，且在微调时表现出通用能力的下降，尤其是小规模模型的分析能力。为解决这些差距，我们引入了ICE-GRT，利用基于近端策略优化（PPO）的人类反馈强化学习（RLHF），展示了在领域内场景中的卓越能力，且不损害通用任务性能。我们对ICE-GRT的探索突显了其理解和推理能力，不仅能生成稳健的答案，还能对答案背后的原因提供详细分析。这一能力标志着超越监督微调模型范围的重大进展。ICE-GRT的成功依赖于几个关键因素，包括适当的数据、奖励规模缩放、KL控制、优势归一化等。ICE-GRT模型在特定领域任务及12项通用语言任务中，相较于同等规模甚至更大规模的LLMs展现出最先进的性能，突显了我们方法的有效性。我们对ICE-GRT进行了全面分析，强调其为LLM领域带来的重大进步。
英文摘要：
The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA encounter limitations in domain-specific tasks, with these models often lacking depth and accuracy in specialized areas, and exhibiting a decrease in general capabilities when fine-tuned, particularly analysis ability in small sized models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization (PPO), demonstrating remarkable ability in in-domain scenarios without compromising general task performance. Our exploration of ICE-GRT highlights its understanding and reasoning ability to not only generate robust answers but also to provide detailed analyses of the reasons behind the answer. This capability marks a significant progression beyond the scope of Supervised Fine-Tuning models. The success of ICE-GRT is dependent on several crucial factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in domain-specific tasks and across 12 general Language tasks against equivalent size and even larger size LLMs, highlighting the effectiveness of our approach. We provide a comprehensive analysis of the ICE-GRT, underscoring the significant advancements it brings to the field of LLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
