面向多样化AI监督的基于原则的探索
Exploration with Principles for Diverse AI Supervision
AI总结：
提出Exploratory AI (EAI)新范式，利用大型语言模型在自然语言空间内自主探索生成高质量训练数据，通过actor-critic架构显著提升模型复杂推理性能，减少对人类监督的依赖。
摘要：
使用下一词预测训练大型transformers带来了AI领域的开创性进展。尽管这种生成式AI方法取得了令人瞩目的成果，但它严重依赖人类监督。即使是像ChatGPT这样最先进的AI模型也依赖于通过人类示范进行微调，需要大量的人类输入和领域专业知识。这种对人类监督的强烈依赖构成了AI创新发展的重大障碍。为了解决这一局限性，我们提出了一种称为Exploratory AI (EAI)的新范式，旨在自主生成高质量训练数据。受无监督强化学习(RL)预训练启发，EAI在自然语言空间内实现探索。我们通过利用大型语言模型来评估生成内容的新颖性来实现这一点。我们的方法采用两个关键组件：一个遵循探索原则生成新颖内容的actor，以及一个评估生成内容并提供critique以指导actor的critic。实证评估表明，EAI显著提升了模型在复杂推理任务上的性能，解决了依赖大量人类监督的局限性。
英文摘要：
Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. Even state-of-the-art AI models like ChatGPT depend on fine-tuning through human demonstrations, demanding extensive human input and domain expertise. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. To address this limitation, we propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data. Drawing inspiration from unsupervised reinforcement learning (RL) pretraining, EAI achieves exploration within the natural language space. We accomplish this by harnessing large language models to assess the novelty of generated content. Our approach employs two key components: an actor that generates novel content following exploration principles and a critic that evaluates the generated content, offering critiques to guide the actor. Empirical evaluations demonstrate that EAI significantly boosts model performance on complex reasoning tasks, addressing the limitations of human-intensive supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
