边缘端多模态Transformer微调：一种并行分割学习方法
Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach
AI总结：
针对多模态Transformer在边缘设备上微调的计算瓶颈，提出并行分割学习方法MPSL，通过轻量级客户端分词器和统一编码器，在7个数据集上匹配或超越联邦学习，减少250倍客户端计算并提升通信可扩展性。
摘要：
多模态Transformer整合了图像、音频和文本等多样数据类型，推动了音视频理解和图像-文本检索等任务的发展；然而，其高参数化限制了在资源受限的边缘设备上的部署。分割学习（Split Learning, SL）在指定的切割层对模型进行划分，将计算密集型操作卸载到服务器，为多模态Transformer的分布式训练提供了一种有前景的方法，但其应用仍未得到充分探索。我们提出了MPSL，一种并行SL方法，用于以分布式方式对多模态Transformer进行计算高效的微调，同时消除了标签共享、客户端同步和每客户端子模型管理。MPSL采用轻量级客户端分词器和统一的模态无关编码器，能够灵活适应特定任务需求。我们在7个多模态数据集上的评估表明，MPSL匹配或优于联邦学习，将客户端计算量减少了250倍，并在通信成本随模型增长方面实现了卓越的可扩展性。通过广泛分析，我们强调了任务适用性、权衡以及MPSL表现出色的场景，以激发进一步探索。
英文摘要：
Multimodal transformers integrate diverse data types like images, audio, and text, advancing tasks such as audio-visual understanding and image-text retrieval; yet their high parameterization limits deployment on resource-constrained edge devices. Split Learning (SL), which partitions models at a designated cut-layer to offload compute-intensive operations to the server, offers a promising approach for distributed training of multimodal transformers, though its application remains underexplored. We present MPSL, a parallel SL approach for computational efficient fine-tuning of multimodal transformers in a distributed manner, while eliminating label sharing, client synchronization, and per-client sub-model management. MPSL employs lightweight client-side tokenizers and a unified modality-agnostic encoder, allowing flexible adaptation to task-specific needs. Our evaluation across 7 multimodal datasets demonstrates that MPSL matches or outperforms Federated Learning, reduces client-side computations by 250x, and achieves superior scalability in communication cost with model growth. Through extensive analysis, we highlight task suitability, trade-offs, and scenarios where MPSL excels, inspiring further exploration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
