理解受污染混合专家模型中极小极大参数估计的专家结构
Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts
AI总结：
本文分析了受污染混合专家模型中的参数估计收敛性，针对提示学习中的提示消失和参数交互减速问题，提出可区分性条件并研究多种专家结构，给出了综合收敛速率与极小极大下界，并通过数值实验验证了理论。
摘要：
我们对受污染混合专家模型中的参数估计进行了收敛性分析。该模型受启发于提示学习问题，在该问题中，人们利用提示（可被建模为专家）来微调大规模预训练模型，以学习下游任务。分析中出现了两个根本性挑战：(i) 预训练模型和提示在混合中的比例可能在训练期间收敛至零，导致提示消失问题；(ii) 预训练模型和提示参数之间的代数交互可能通过某些偏微分方程发生，并减缓提示学习。作为应对，我们引入了可区分性条件来控制前述参数交互。此外，我们还研究了多种类型的专家结构，以理解它们对参数估计收敛行为的影响。在每种场景下，我们提供了参数估计的综合收敛速率以及相应的极小极大下界。最后，我们运行了若干数值实验，从经验上验证了我们的理论发现。
英文摘要：
We conduct the convergence analysis of parameter estimation in the contaminated mixture of experts. This model is motivated from the prompt learning problem where ones utilize prompts, which can be formulated as experts, to fine-tune a large-scale pre-trained model for learning downstream tasks. There are two fundamental challenges emerging from the analysis: (i) the proportion in the mixture of the pre-trained model and the prompt may converge to zero during the training, leading to the prompt vanishing issue; (ii) the algebraic interaction among parameters of the pre-trained model and the prompt can occur via some partial differential equations and decelerate the prompt learning. In response, we introduce a distinguishability condition to control the previous parameter interaction. Additionally, we also investigate various types of expert structure to understand their effects on the convergence behavior of parameter estimation. In each scenario, we provide comprehensive convergence rates of parameter estimation along with the corresponding minimax lower bounds. Finally, we run several numerical experiments to empirically justify our theoretical findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
