PrivFedTalk: 隐私感知的联邦扩散模型与身份稳定的适配器用于个性化说话头生成
PrivFedTalk: Privacy-Aware Federated Diffusion with Identity-Stable Adapters for Personalized Talking-Head Generation
AI总结：
本文提出PrivFedTalk框架，结合条件潜在扩散模型与参数高效的身份适应，解决个性化说话头生成中的隐私问题，通过身份稳定联邦聚合和时间去噪一致性正则化提升性能。
摘要：
说话头生成随着基于扩散的生成模型快速发展，但训练通常依赖于集中化的面部视频和语音数据集，引发重大隐私问题。对于个性化说话头生成，身份特定的数据高度敏感且通常无法跨用户或设备聚合。PrivFedTalk是一种隐私感知的联邦框架，结合条件潜在扩散与参数高效的识别人适配器。一个共享的扩散主干在客户端上训练，而每个客户端从本地私有音频视频数据中学习轻量级LoRA身份适配器，避免原始数据共享并减少通信成本。为应对异构客户端分布，身份稳定的联邦聚合（ISFA）使用隐私安全的标量可靠性信号对客户端更新进行加权，该信号由设备上的身份一致性和时间稳定性估计计算得出。引入时间去噪一致性（TDC）正则化以减少联邦去噪过程中的帧漂移、闪烁和身份漂移。为限制更新侧的隐私风险，安全聚合和客户端级差分隐私应用于适配器更新。实现支持低内存GPU执行和异构共享硬件上的多GPU客户端并行训练。在多种训练和聚合条件下与PrivFedTalk、FedAvg和FedProx的比较实验显示，稳定了联邦优化并成功在受限资源下完成了端到端训练和评估。结果支持在联邦环境中隐私感知的个性化说话头训练的可行性，同时表明需要进一步标准化评估以强化组件级、隐私-效用和定性主张。
英文摘要：
Talking-head generation has advanced rapidly with diffusion-based generative models, but training usually depends on centralized face-video and speech datasets, raising major privacy concerns. The problem is more acute for personalized talking-head generation, where identity-specific data are highly sensitive and often cannot be pooled across users or devices. PrivFedTalk is presented as a privacy-aware federated framework for personalized talking-head generation that combines conditional latent diffusion with parameter-efficient identity adaptation. A shared diffusion backbone is trained across clients, while each client learns lightweight LoRA identity adapters from local private audio-visual data, avoiding raw data sharing and reducing communication cost. To address heterogeneous client distributions, Identity-Stable Federated Aggregation (ISFA) weights client updates using privacy-safe scalar reliability signals computed from on-device identity consistency and temporal stability estimates. Temporal-Denoising Consistency (TDC) regularization is introduced to reduce inter-frame drift, flicker, and identity drift during federated denoising. To limit update-side privacy risk, secure aggregation and client-level differential privacy are applied to adapter updates. The implementation supports both low-memory GPU execution and multi-GPU client-parallel training on heterogeneous shared hardware. Comparative experiments on the present setup across multiple training and aggregation conditions with PrivFedTalk, FedAvg, and FedProx show stable federated optimization and successful end-to-end training and evaluation under constrained resources. The results support the feasibility of privacy-aware personalized talking-head training in federated environments, while suggesting that stronger component-wise, privacy-utility, and qualitative claims need further standardized evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
