Dyad：为大型语言模型扩展原生类型化决策能力

中文摘要 AI 辅助

我们研究如何通过为大型语言模型（LLM）扩展原生类型化决策能力来构建更强大的通用智能体。我们提出了Dyad，一种架构，它通过一个环境条件化的动作编码器增强预训练的LLM，该编码器并行嵌入每个候选动作描述，然后将这些嵌入与LLM的内部状态进行评分，以产生类型化动作上的分布。通过将决策分解为演化交互状态的表示和环境特定的动作语义，Dyad引入了一种归纳偏置，用于学习可复用的表示，同时即使在动作空间增长时也能保持动作评分的效率。我们研究了由环境交互驱动的两种互补的强化学习设置。在LLM冻结的情况下，仅训练动作编码器在四个未见环境中获得了一致的性能提升，实现了无需修改任何LLM参数的模块化适应。联合优化两个组件在多样化的交互任务和模型规模上优于传统的RL后训练，包括在9B模型上ALFWorld平均绝对提升3.80%，同时提高了通用知识、推理和编码能力。

英文摘要

We study how to build more capable general-purpose agents by extending large language models (LLMs) with native typed decision-making. We introduce Dyad, an architecture that augments a pretrained LLM with an environment-conditioned action encoder that embeds each candidate action description in parallel, then scores these embeddings against the LLM's internal state to yield a distribution over typed actions. By factorizing decision-making into representations of the evolving interaction state and environment-specific action semantics, Dyad introduces an inductive bias for learning reusable representations while keeping action scoring efficient even as the action space grows. We investigate two complementary reinforcement learning settings driven by environment interaction. With the LLM frozen, training the action encoder alone achieves consistent gains across four unseen environments, enabling modular adaptation without modifying any LLM parameters. Jointly optimizing both components outperforms conventional RL post-training across diverse interactive tasks and model scales, including a 3.80% average absolute gain on ALFWorld with a 9B model, while improving general knowledge, reasoning, and coding.

发表机构

• Zhejiang University（浙江大学）
• SAP（SAP公司）
• Central South University（中南大学）
• The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
• Chalmers University of Technology（查尔姆斯理工大学）
• Microsoft Research（微软研究院）

机构由 AI 辅助整理，请以论文原文为准。