开放智能体系统中多智能体强化学习信用分配的挑战
Challenges in Credit Assignment for Multi-Agent Reinforcement Learning in Open Agent Systems
AI总结：
本文研究了开放多智能体强化学习中的信用分配问题，通过概念分析与实证研究揭示了智能体和任务的动态开放性会破坏传统假设，导致信用错误归因和性能显著下降。
摘要：
在快速发展的多智能体强化学习（MARL）领域，理解开放系统的动态特性至关重要。MARL中的开放性是指系统中智能体群体、任务和智能体类型的动态特性。具体而言，如(Eck et al. 2023) [2]所述，存在三种类型的开放性：智能体开放性，即智能体可以随时进入或离开系统；任务开放性，即新任务出现，现有任务演变或消失；以及类型开放性，即智能体的能力和行为随时间变化。本报告提供了概念和实证综述，重点关注开放性与信用分配问题（CAP）之间的相互作用。CAP涉及确定单个智能体对整体系统性能的贡献，这一任务在开放环境中变得日益复杂。传统信用分配（CA）方法通常假设智能体群体是静态的、任务是固定且预定义的、类型是平稳的，这使得它们不适用于开放系统。我们首先进行概念分析，引入开放性的新子类别，以详细说明智能体流动或任务取消等事件如何打破支撑现有CAP方法的环境平稳性和固定团队组成假设。然后，我们在开放环境中使用具有代表性的时间和结构算法进行实证研究。结果表明，开放性直接导致信用错误归因，其表现为不稳定的损失函数和显著的性能下降。
英文摘要：
In the rapidly evolving field of multi-agent reinforcement learning (MARL), understanding the dynamics of open systems is crucial. Openness in MARL refers to the dynam-ic nature of agent populations, tasks, and agent types with-in a system. Specifically, there are three types of openness as reported in (Eck et al. 2023) [2]: agent openness, where agents can enter or leave the system at any time; task openness, where new tasks emerge, and existing ones evolve or disappear; and type openness, where the capabil-ities and behaviors of agents change over time. This report provides a conceptual and empirical review, focusing on the interplay between openness and the credit assignment problem (CAP). CAP involves determining the contribution of individual agents to the overall system performance, a task that becomes increasingly complex in open environ-ments. Traditional credit assignment (CA) methods often assume static agent populations, fixed and pre-defined tasks, and stationary types, making them inadequate for open systems. We first conduct a conceptual analysis, in-troducing new sub-categories of openness to detail how events like agent turnover or task cancellation break the assumptions of environmental stationarity and fixed team composition that underpin existing CAP methods. We then present an empirical study using representative temporal and structural algorithms in an open environment. The results demonstrate that openness directly causes credit misattribution, evidenced by unstable loss functions and significant performance degradation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
