SOCK：一种衡量大语言模型自我复制能力的基准
SOCK: A Benchmark for Measuring Self-Replication in Large Language Models
AI总结：
SOCK提出了一种评估大语言模型自我复制能力的基准，旨在建立标准并减少多代理系统中的潜在风险。
摘要：
我们介绍了SOCK，一种用于衡量大语言模型（LLM）在没有人工干预的情况下自我复制能力的基准命令行接口（CLI）。在该基准中，自我复制不仅指LLM能够创建一个功能正常且运行的自身副本，还包括该自我复制在不同计算环境中持续发生的能力。相应地，我们开发了一个系统，根据LLM在自我复制方面的广泛能力将其分为两大类别：自我复制能力等级（RCL）和持续性能力等级（PCL）。使用基于实际可操作的现代CLI工具和计算机进程的五项任务套件，实验在受控环境中通过LLM作为主动代理进行编排。LLM在代理任务上的表现被计算以生成R分数（整体自我复制能力的定量评估），并利用数据将LLM分类到特定的RCL-PCL矩阵中。SOCK有两个主要贡献：（1）提供了评估LLM自我复制能力的第一个正式定义和基准套件，旨在建立未来研究的标准；（2）允许行业跟踪未来多代理系统的效果，并减轻其中潜在的自我复制威胁向量。从评估各种开源和专有前沿模型的结果中可以看出，持续自我复制和多代理系统存在显著障碍，包括上下文保留和多代理决策制定。我们提出未来研究方向以安全地减少这些障碍的严重性， potentially降低未来更功能性的多代理系统的风险。
英文摘要：
We introduce SOCK, a benchmark command line interface (CLI) that measures large language models' (LLMs) ability to self-replicate without human intervention. In this benchmark, self-replication is defined not only as an LLM's ability to create a functioning and running copy of itself, but also the ability for that self-replication to persist and occur across different computational contexts. Accordingly, we've developed a system to categorize LLMs based on broad self-replication capabilities in two general classes, Replication-Capability Levels (RCL) and Persistence-Capability Levels (PCL). Using a five-task suite based on practically manipulable modern CLI utilities and computer processes, experiments are orchestrated in a controlled environment with an LLM acting agentically. The performance of the LLM on agent tasks is then computed to produce an R-score (a quantitative evaluation of overall self-replication ability) and data used to categorize LLMs into specific RCL-PCL matrices. SOCK offers two primary contributions: (1) Provides the first formalized definitions and benchmark suite for evaluating LLM self-replication, with the goal of establishing a standard for future research, to our knowledge; (2) Allows the industry to track the effectiveness of future multi-agent systems and mitigate potential self-replication threat vectors within them. The results compiled from evaluating a variety of open-weight and proprietary frontier models reveal significant obstacles to persistent self-replication and multi-agent systems, including context retention and multi-agent decision-making. We propose future research directions to safely reduce the severity of these obstacles, potentially lowering future risk of more functional multi-agent systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
