CoGenesis：一个协同大语言模型与小语言模型以实现安全上下文感知指令跟随的框架
CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following
AI总结：
本文提出 CoGenesis 框架，通过云端大模型与本地小模型协同生成，在保护用户隐私的同时实现上下文感知的指令跟随，实验证明其具有竞争力。
摘要：
随着语言模型（LMs）的发展，其接触私有数据的情况日益不可避免，并且它们在个人设备（如个人电脑和智能手机）上的部署（尤其是较小模型）已成为普遍趋势。在充满用户信息的上下文中，使模型既能保护用户隐私又能高效执行命令，成为一项重要的研究需求。本文提出 CoGenesis，一个协同生成框架，将大型模型（托管在云基础设施上）与小型模型（部署在本地设备上）相结合，以从逻辑上应对隐私问题。首先，我们设计了一个流程，用于创建富含广泛上下文细节的个性化写作指令数据集，作为该研究问题的测试平台。随后，我们分别基于草图（sketch）和逻辑值（logits）引入了 CoGenesis 的两种变体。基于我们合成的数据集以及另外两个开源数据集的实验结果表明：1）大规模模型在提供用户上下文时表现良好，但在缺乏此类上下文时表现挣扎。2）尽管在合成数据集上微调的专用小型模型显示出潜力，但仍落后于其更大的对应模型。3）我们的 CoGenesis 框架利用混合规模模型，展示了具有竞争力的性能，为隐私问题提供了一种可行的解决方案。
英文摘要：
With the advancement of language models (LMs), their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, such as PCs and smartphones, has become a prevailing trend. In contexts laden with user information, enabling models to both safeguard user privacy and execute commands efficiently emerges as an essential research imperative. In this paper, we propose CoGenesis, a collaborative generation framework integrating large (hosted on cloud infrastructure) and small models (deployed on local devices) to address privacy concerns logically. Initially, we design a pipeline to create personalized writing instruction datasets enriched with extensive context details as the testbed of this research issue. Subsequently, we introduce two variants of CoGenesis based on sketch and logits respectively. Our experimental findings, based on our synthesized dataset and two additional open-source datasets, indicate that: 1) Large-scale models perform well when provided with user context but struggle in the absence of such context. 2) While specialized smaller models fine-tuned on the synthetic dataset show promise, they still lag behind their larger counterparts. 3) Our CoGenesis framework, utilizing mixed-scale models, showcases competitive performance, providing a feasible solution to privacy issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
