去中心化扩散模型（DDMs）通过在不同数据簇上独立训练的专家进行去噪，这些专家的预测可能强烈不一致。在这样的系统中，什么决定了生成的质量？我们首次系统地探讨了这个问题。事先预期最小化去噪轨迹敏感度——最小化在采样过程中扰动的放大程度——应决定生成质量。我们证明这一假设是错误的：存在稳定性与质量的分离。全集路由，在每一步都结合所有专家的预测，实现了最稳定的采样动态和最佳的数值收敛，但生成质量最差（FID 47.9 vs. 22.6 for sparse Top-2 routing）。相反，我们确定专家-数据对齐是主导原则：生成质量取决于输入到专家的路由，这些专家的训练分布覆盖当前去噪状态。在两个不同的DDM系统中，我们通过（i）数据簇距离分析，确认稀疏路由选择与当前去噪状态最接近的数据簇的专家；（ii）每个专家分析，显示所选专家比非所选专家产生更准确的预测；（iii）专家分歧分析，显示当专家意见不一致时质量下降。对于DDM部署，我们的发现确立了路由应优先考虑专家-数据对齐而非数值稳定性指标。
英文摘要：
Decentralized Diffusion Models (DDMs) route denoising through experts trained independently on disjoint data clusters, which can strongly disagree in their predictions. What governs the quality of generations in such systems? We present the first ever systematic investigation of this question. A priori, the expectation is that minimizing denoising trajectory sensitivity -- minimizing how perturbations amplify during sampling -- should govern generation quality. We demonstrate this hypothesis is incorrect: a stability-quality dissociation. Full ensemble routing, which combines all expert predictions at each step, achieves the most stable sampling dynamics and best numerical convergence while producing the worst generation quality (FID 47.9 vs. 22.6 for sparse Top-2 routing). Instead, we identify expert-data alignment as the governing principle: generation quality depends on routing inputs to experts whose training distribution covers the current denoising state. Across two distinct DDM systems, we validate expert-data alignment using (i) data-cluster distance analysis, confirming sparse routing selects experts with data clusters closest to the current denoising state, and (ii) per-expert analysis, showing selected experts produce more accurate predictions than non-selected ones, and (iii) expert disagreement analysis, showing quality degrades when experts disagree. For DDM deployment, our findings establish that routing should prioritize expert-data alignment over numerical stability metrics.
dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；PKU-OCTA Laboratory for Blockchain and Privacy Computing（区块链与隐私计算实验室）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心）；State University of New York at Buffalo（纽约州立大学布法罗分校）
Many recent breakthroughs in deep learning were achieved by training increasingly larger models on massive datasets. However, training such models can be prohibitively expensive. For instance, the cluster used to train GPT-3 costs over \$250 million. As a result, most researchers cannot afford to train state of the art models and contribute to their development. Hypothetically, a researcher could crowdsource the training of large neural networks with thousands of regular PCs provided by volunteers. The raw computing power of a hundred thousand \$2500 desktops dwarfs that of a \$250M server pod…
GRASP Laboratory, University of Pennsylvania（宾夕法尼亚大学GRASP实验室）；Department of Electrical and Computer Engineering, Drexel University（德雷塞尔大学电气与计算机工程系）
In this work, we study the problem of decentralized multi-agent perimeter defense that asks for computing actions for defenders with local perceptions and communications to maximize the capture of intruders. One major challenge for practical implementations is to make perimeter defense strategies scalable for large-scale problem instances. To this end, we leverage graph neural networks (GNNs) to develop an imitation learning framework that learns a mapping from defenders' local perceptions and their communication graph to their actions. The proposed GNN-based learning network is trained by imi…
Most existing federated learning methods assume that clients have fully labeled data to train on, while in reality, it is hard for the clients to get task-specific labels due to users' privacy concerns, high labeling costs, or lack of expertise. This work considers the server with a small labeled dataset and intends to use unlabeled data in multiple clients for semi-supervised learning. We propose a new framework with a generalized model, Federated Incremental Learning (FedIL), to address the problem of how to utilize labeled data in the server and unlabeled data in clients separately in the s…
Meta-learning usually refers to a learning algorithm that learns from other learning algorithms. The problem of uncertainty in the predictions of neural networks shows that the world is only partially predictable and a learned neural network cannot generalize to its ever-changing surrounding environments. Therefore, the question is how a predictive model can represent multiple predictions simultaneously. We aim to provide a fundamental understanding of learning to learn in the contents of Decentralized Neural Networks (Decentralized NNs) and we believe this is one of the most important questio…
