arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.05060cs.AIcs.MAcs.SI
收藏

社会网络结构与创新--RL与SAPIENS的经验分享

Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS

Inria(法国国家信息与自动化研究所);Ensta ParisTech(巴黎高等先进技术学院);University Pompeu Fabra(庞培法布拉大学);Microsoft Research(微软研究院)

Eleni Nisioti, Mateo Mahaut, Pierre-Yves Oudeyer, Ida Momennejad, Clément Moulin-Frier

摘要:

人类文化依赖于创新:我们不断探索如何将现有元素结合起来创造新元素的能力。创新不是单打独斗,它依赖于集体的探索和积累。强化学习(RL)方法通常假设完全连接的群体最适合创新。然而,人类的实验室和实地研究表明,层次创新更稳健地实现了动态的社会网络结构。在动态环境中,人类在单独创新或小规模集群创新,然后与他人分享成果之间摇摆不定。据我们所知,社会网络结构对创新的作用在强化学习中还没有系统的研究。在这里,我们使用一个多层次的问题设置(WordCraft),三个不同的创新任务来测试的假设,社会网络结构影响分布式RL算法的性能。我们系统地设计了DQN网络,这些DQN在不同的结构(全连接、小世界、动态、环形)中从重放缓冲区中分享经验,并引入了一组行为和助记符指标,这些指标扩展了RL的经典奖励评估框架。通过比较不同社会网络结构在不同任务中的创新水平表明,首先,与人类的发现一致,在具有欺骗性和大搜索空间的任务中,动态结构中的经验共享实现了最高水平的创新。第二,如果只有一条明确的创新道路,经验分享就没有那么大的帮助。第三,我们提出的指标可以帮助理解不同社交网络结构在不同任务上的成功,个人和团体层面的经验多样性提供了重要的见解。

英文摘要:

Human culture relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones. Innovation is not solitary, it relies on collective search and accumulation. Reinforcement learning (RL) approaches commonly assume that fully-connected groups are best suited for innovation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures. In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role of social network structure on innovation has not been systematically studied in RL. Here, we use a multi-level problem setting (WordCraft), with three different innovation tasks to test the hypothesis that the social network structure affects the performance of distributed RL algorithms. We systematically design networks of DQNs sharing experiences from their replay buffers in varying structures (fully-connected, small world, dynamic, ring) and introduce a set of behavioral and mnemonic metrics that extend the classical reward-focused evaluation framework of RL. Comparing the level of innovation achieved by different social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation in tasks with a deceptive nature and large search spaces. Second, experience sharing is not as helpful when there is a single clear path to innovation. Third, the metrics we propose, can help understand the success of different social network structures on different tasks, with the diversity of experiences on an individual and group level lending crucial insights.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑