ML-ECS: 一种用于边缘-云协同的协同多模态学习框架
ML-ECS: Heterogeneity-Aware Collaborative Multimodal Learning for Edge-Cloud Synergies
AI总结:
ML-ECS提出了一种协同多模态学习框架,通过跨模态对比学习、自适应多模态调节、模态感知聚合和SLM增强的CCL,实现边缘-云协同中的高效知识转移与性能提升。
摘要:
边缘-云协同提供了一种有前景的隐私保护基础模型部署范式,其中轻量级设备模型适应于领域特定的数据,而云托管模型协调知识共享。然而,在现实世界边缘环境中,协同多模态学习面临模态异质性(不同领域中的不同模态组合)和模型结构异质性(不同模态特定的编码器/融合模块)的挑战。为了解决这些问题,我们提出了ML-ECS,一种协同多模态学习框架,该框架能够实现服务器端模型与异构边缘模型的联合训练。该框架由四个组成部分组成:(1)跨模态对比学习(CCL)以在共享潜在空间中对齐模态表示,(2)自适应多模态调节(AMT)以保留本地数据集中的领域特定知识,(3)模态感知模型聚合(MMA)以在缓解因缺失模态引起的噪声的同时进行稳健聚合,以及(4)SLM增强的CCL(SE-CCL)以促进云和边缘之间的双向知识转移。在各种多模态任务上的实验结果表明,ML-ECS在不同模态可用性下始终优于最先进的基线,在Rouge-LSum上实现了5.44%到12.08%的改进,并提升了客户端和服务器端的性能。此外,通过仅通信低秩LoRA参数和融合表示,ML-ECS实现了高效的通信效率,仅需总参数体积的0.65%。
英文摘要:
Edge-cloud synergies provide a promising paradigm for privacy-preserving deployment of foundation models in heterogeneous web environments, enabling lightweight edge models to support domain-specific multimodal we b services while cloud-hosted models facilitate collaborative knowledge sharing. However, in practical edge-cloud web environments, collaborative multimodal learning is challenged by modality heterogeneity (different modality combinations across domains) and model-structure heterogeneity (different modality-specific encoders/fusion modules). To address these issues, we propose ML-ECS, a collaborative multimodal learning framework that enables joint training between a server-based model and heterogeneous edge models. This framework consists of four components: (1) cross-modal contrastive learning (CCL) to align modality representations in a shared latent space, (2) adaptive multimodal tuning (AMT) to preserve domain-specific knowledge from local datasets, (3) modality-aware aggregation (MAA) to robustly aggregate while mitigating noise caused by missing modalities, and (4) SLM-enhanced CCL (SE-CCL) to facilitate bidirectional knowledge transfer between cloud and edge. Experimental results on various multimodal tasks show that ML-ECS consistently outperform state-of-the-art baselines under varying modality availability, achieving improvements of 1.55% to 16.75% in Rouge-LSum and improving both client- and server-side performance. In addition, by communicating only low-rank LoRA parameters and fused representations, ML-ECS achieves high communication efficiency, requiring only 0.71% of the total parameter volume.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。