arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.05974cs.LGcs.CL
收藏

面向移动 NLP 的联邦少样本学习

Federated Few-Shot Learning for Mobile NLP

Beiyou Shenzhen Institute(北邮深圳研究院);University of Virginia(弗吉尼亚大学)

Dongqi Cai, Shangguang Wang, Yaozong Wu, Felix Xiaozhu Lin, Mengwei Xu

AI总结:

本文研究移动 NLP 中联邦少样本学习问题,结合伪标签与提示学习,并通过 FeS 的课程节奏、数据选择和模型容量协同优化,在极少标注下保持准确率且大幅降低训练开销。

摘要:

自然语言处理(NLP)在移动设备上有着丰富的应用。为支持各类语言理解任务,基础 NLP 模型通常会在联邦、隐私保护的设置(FL)中进行微调。这一过程目前依赖来自移动客户端的至少数十万条带标签训练样本;然而移动用户往往缺乏标注数据的意愿或知识。这种数据标签不足的情况被称为少样本场景,并成为移动 NLP 应用的关键阻碍。\n本研究首次考察少样本场景下的联邦 NLP(FedFSL)。通过改造伪标签和提示学习方面的算法进展,我们首先建立了一条训练流水线:当仅有 0.05%(少于 100 条)训练数据被标注、其余数据均无标签时,该流水线仍能取得具有竞争力的准确率。为实例化这一工作流,我们进一步提出系统 FeS,通过新颖设计解决高昂执行成本问题:(1)课程节奏,即以与学习进度相匹配的速率将伪标签注入训练工作流;(2)表征多样性,这是一种选择最具可学习性数据的机制,并且只会为这些数据生成伪标签;(3)对模型训练深度和层容量进行协同规划。这些设计共同将训练延迟、客户端能耗和网络流量分别降低至多 46.0×、41.2× 和 3000.0×。通过算法与系统协同设计,FFNLP 证明 FL 可应用于大多数训练样本均无标签的挑战性设置。

英文摘要:

Natural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least hundreds of thousands of labeled training samples from mobile clients; yet mobile users often lack willingness or knowledge to label their data. Such an inadequacy of data labels is known as a few-shot scenario; it becomes the key blocker for mobile NLP applications. For the first time, this work investigates federated NLP in the few-shot scenario (FedFSL). By retrofitting algorithmic advances of pseudo labeling and prompt learning, we first establish a training pipeline that delivers competitive accuracy when only 0.05% (fewer than 100) of the training data is labeled and the remaining is unlabeled. To instantiate the workflow, we further present a system FeS, addressing the high execution cost with novel designs. (1) Curriculum pacing, which injects pseudo labels to the training workflow at a rate commensurate to the learning progress; (2) Representational diversity, a mechanism for selecting the most learnable data, only for which pseudo labels will be generated; (3) Co-planning of a model's training depth and layer capacity. Together, these designs reduce the training delay, client energy, and network traffic by up to 46.0$\times$, 41.2$\times$ and 3000.0$\times$, respectively. Through algorithm/system co-design, FFNLP demonstrates that FL can apply to challenging settings where most training samples are unlabeled.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑