光线追踪专家混合
Mixture of Raytraced Experts
AI总结：
提出光线追踪专家混合架构，通过动态选择专家序列生成可变计算图，无需负载均衡即可在减少训练轮数的同时提升精度。
摘要：
我们引入了光线追踪专家混合，这是一种堆叠的混合专家架构，能够动态选择专家序列，从而生成具有可变宽度和深度的计算图。现有的MoE架构通常对给定样本需要固定计算量。相比之下，我们的方法随着计算在专家序列中循环，产生的预测精度不断提高。我们通过从一组候选专家中迭代采样来训练模型，展开序列类似于循环神经网络的训练方式。该方法不需要负载均衡机制，初步实验表明，在具有可比或更高精度的情况下，训练轮数减少了10%至40%。这些结果指出了MoE领域的新研究方向，允许设计潜在更快且更具表达能力的模型。代码可在https://github.com/nutig/RayTracing获取。
英文摘要：
We introduce a Mixture of Raytraced Experts, a stacked Mixture of Experts (MoE) architecture which can dynamically select sequences of experts, producing computational graphs of variable width and depth. Existing MoE architectures generally require a fixed amount of computation for a given sample. Our approach, in contrast, yields predictions with increasing accuracy as the computation cycles through the experts' sequence. We train our model by iteratively sampling from a set of candidate experts, unfolding the sequence akin to how Recurrent Neural Networks are trained. Our method does not require load-balancing mechanisms, and preliminary experiments show a reduction in training epochs of 10\% to 40\% with a comparable/higher accuracy. These results point to new research directions in the field of MoEs, allowing the design of potentially faster and more expressive models. The code is available at https://github.com/nutig/RayTracing
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
