Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
In this paper, we explore a novel federated multimodal instruction tuning task(FedMIT), which is significant for collaboratively fine-tuning MLLMs on different types of multimodal instruction data on distributed devices. To solve the new task, we propose a federated multimodal instruction tuning framework(Pilot). Our framework integrates two stages of "adapter on adapter" into the connector of the vision encoder and the LLM. In stage 1, we extract task-specific features and client-specific features from visual information. In stage 2, we build the cross-task Mixture-of-Adapters(CT-MoA) module to perform cross-task interaction. Each client can not only capture personalized information of local data and learn task-related multimodal information, but also learn general knowledge from other tasks. In addition, we introduce an adaptive parameter aggregation strategy for text training parameters, which optimizes parameter aggregation by calculating weights based on the euclidean distance between parameters, so that parameter aggregation can benefit from positive effects to the greatest extent while effectively reducing negative effects. Our framework can collaboratively exploit distributed data from different local clients to learn cross-task knowledge without being affected by the task heterogeneity during instruction tuning. The effectiveness of our method is verified in two different cross-task scenarios.
DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition
Seltasquare Seoul, Korea（首尔韩国Seltasquare）；Seoul National University College of Medicine（首尔国立大学医学院）；Biomedical Research Institute, Seoul National University Hospital（首尔国立大学医院生物医学研究所）
Offscript: Automated Auditing of Instruction Adherence in LLMs
University of Washington Information School Seattle Washington USA；University of Washington\ G. Allen School of Computer Science \& Engineering Seattle Washington USA；University of Washington Information School；University of Washington\ G. Allen School of Computer Science \& Engineering
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Imperial College London（帝国理工学院）；Technische Universität Berlin (TU Berlin)（柏林工业大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kuaishou Technology（快手科技）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人创新中心）
Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles
University of California Merced（加州大学默塞德分校）；Aalborg University（奥尔堡大学）
School of Advanced Interdisciplinary Sciences, UCAS（中国科学院大学先进交叉学科学院）；MAIS, CASIA（中国科学院自动化所人工智能研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港科技大学人工智能与机器人中心，中国科学院）
CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Qilu Hospital of Shandong University（山东大学齐鲁医院）；National University o（国立大学）
Secure Multi-Modal Data Fusion in Federated Digital Health Systems via MCP
School of Information, Computer, and Communication Technology（信息、计算机与通信技术学院）；Sirindhorn International Institute of Technology, Thammasat University（泰国朱拉安吞国际技术学院，泰国 Thammasat 大学）
Federated Data Analytics for Cancer Immunotherapy: A Privacy-Preserving Collaborative Platform for Patient Management
Faculty of Computers & Artificial Intelligence, Cairo University（开罗大学计算机与人工智能学院）；Scientific Academy for Service Technology e.V. (ServTech)（服务技术科学学会）；University of New South Wales（新南威尔士大学）；University of Macquarie（麦考瑞大学）；Egypt University of Informatics（埃及信息学大学）
Scalable Cross-Facility Federated Learning for Scientific Foundation Models on Multiple Supercomputers
Mathematics and Computer Science Division, Argonne National Laboratory（阿贡国家实验室数学与计算机科学 division）；Data Science and Learning Division, Argonne National Laboratory（阿贡国家实验室数据科学与学习 division）；Department of Computer Science, University of Chicago（芝加哥大学计算机科学系）
