客户端非独立同分布下的分布式上下文学习
Distributed In-Context Learning under Non-IID Among Clients
AI总结:
本文针对客户端数据非独立同分布下的分布式上下文学习问题,提出一种基于查询偏好为每个客户端数据驱动分配预算的新方法,并通过多数据集实验验证其优于现有基线。
摘要:
大语言模型(LLMs)的进展已显示出其在多个复杂的自然语言推理任务中的有效性。一个关键挑战仍然在于如何高效地将这些模型适配到新的或不熟悉的任务上。上下文学习(ICL)通过从训练数据集中检索一组与查询相关的数据点(称为上下文示例,ICE),并在推理过程中将其作为上下文提供,为少样本适配提供了一种有前景的解决方案。大多数现有研究利用集中式训练数据集,但许多现实世界的数据集可能分布在多个客户端之间,并且远程数据检索可能伴随成本。尤其是当客户端数据为非独立同分布(non-IID)时,从客户端检索测试查询所需的一组合适的ICE面临关键挑战。在本文中,我们首先表明,在这一具有挑战性的设定下,由于非独立同分布性,测试查询在不同客户端之间会有不同的偏好,而等量贡献往往导致次优性能。随后,我们引入一种新方法,以在存在数据使用预算的情况下解决分布式非独立同分布ICL问题。其原理是,每个客户端的适当贡献(预算)应根据每个查询对该客户端的偏好来设计。我们的方法采用数据驱动的方式为每个客户端分配预算,并针对每个测试查询进行定制。通过在多样化数据集上的广泛实证研究,我们的框架相对于竞争基线展示了优越的性能。
英文摘要:
Advancements in large language models (LLMs) have shown their effectiveness in multiple complicated natural language reasoning tasks. A key challenge remains in adapting these models efficiently to new or unfamiliar tasks. In-context learning (ICL) provides a promising solution for few-shot adaptation by retrieving a set of data points relevant to a query, called in-context examples (ICE), from a training dataset and providing them during the inference as context. Most existing studies utilize a centralized training dataset, yet many real-world datasets may be distributed among multiple clients, and remote data retrieval can be associated with costs. Especially when the client data are non-identical independent distributions (non-IID), retrieving from clients a proper set of ICEs needed for a test query presents critical challenges. In this paper, we first show that in this challenging setting, test queries will have different preferences among clients because of non-IIDness, and equal contribution often leads to suboptimal performance. We then introduce a novel approach to tackle the distributed non-IID ICL problem when a data usage budget is present. The principle is that each client's proper contribution (budget) should be designed according to the preference of each query for that client. Our approach uses a data-driven manner to allocate a budget for each client, tailored to each test query. Through extensive empirical studies on diverse datasets, our framework demonstrates superior performance relative to competing baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。