arXiv 2308.032752023-08-07cs.CL

基于适配器的选择性知识蒸馏用于联邦多领域会议摘要

Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization

Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology（哈尔滨工业大学社会计算与信息检索研究中心）；Peng Cheng Laboratory（鹏城实验室）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）

Xiachong Feng, Xiaocheng Feng, Xiyuan Du, Min-Yen Kan, Bing Qin

备注：: This work has been submitted to the IEEE TASLP for possible publication

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对会议摘要中数据敏感、多领域non-IID分布及通信成本高的问题，提出AdaFedSelecKD，通过双适配器与选择性知识蒸馏实现低通信、鲁棒的联邦训练。

摘要：

会议摘要已成为一种可为用户提供精简摘要的有前景技术。然而，现有工作集中于在集中式数据上训练模型，忽视了现实场景：由于会议数据具有敏感性，无法对其进行集中收集。这一差距促使我们探索用于会议摘要的联邦学习。有两个关键挑战阻碍了进展。首先，最先进的摘要器基于参数量庞大的预训练模型。在客户端之间交换此类模型的参数会带来巨大的带宽成本。其次，由于现实世界中的会议数据属于不同领域并分布在各个客户端，它们是非独立同分布（non-IID）的实例。IID假设并不成立，这会改变哪些形式的学习算法最适用。为解决该问题，我们提出了基于适配器的联邦选择性知识蒸馏（AdaFedSelecKD），用于训练高性能的客户端模型。具体而言，我们开发了一种基于适配器的摘要模型，其中两个适配器协同工作，以更少的参数促进学习，从而降低通信成本。随后，我们设计了一种选择性知识蒸馏策略，帮助客户端在基于non-IID数据利用全局参数的同时，稳健地处理其自身数据上的领域聚焦建模。在QMSum基准上进行的大量实验表明，AdaFedSelecKD能够取得与强大的集中式训练方法相当的性能，并展现出其泛化能力和鲁棒性。

英文摘要：

Meeting summarization has emerged as a promising technique for providing users with condensed summaries. However, existing work has focused on training models on centralized data, neglecting real-world scenarios where meeting data are infeasible to collect centrally, due to their sensitive nature. This gap motivates us to explore federated learning for meeting summarization. Two critical challenges impede progress. First, state-of-the-art summarizers are based on parameter-heavy pre-trained models. Exchanging such a model's parameters across clients imposes large bandwidth costs. Second, as real-world meeting data belong to various domains and are distributed across clients, they are instances of non-identically and independently distributed (non-IID). IID assumptions do not hold, which changes which forms of learning algorithms best apply. To address this, we propose Adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models. Specifically, we develop an adapter-based summarization model where two adapters cooperatively facilitate learning using fewer parameters to reduce communication costs. Then, we devise a selective knowledge distillation strategy, assisting clients in robustly handling domain-focused modelling on their own data, while leveraging global parameters based on non-IID data. Extensive experiments on the QMSum benchmark demonstrate AdaFedSelecKD can achieve comparable performance with powerful centralized training methods, and shows its generalizability and robustness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。