训练期间的稀疏竞争促进专用模块的出现
Sparse Competition during Training For the Emergence of Specialized Modules
浏览论文内容
中文总结 AI 辅助
该研究提出一种稀疏竞争方法，无需模块级监督即可在ImageNet-100和CIFAR-100上诱导深度神经网络的功能模块化，使专用模块出现并捕获数据的高级结构。
中文摘要 AI 辅助
深度神经网络的模块化已被提出作为通过促进解耦表示和减少冗余来提升可解释性与训练效果的手段。本研究探讨训练期间神经元组间的竞争动力学如何促使模块化结构出现，我们提出一种方法，该方法：（i）保持接近基线的准确率；（ii）通过将输入稀疏路由到神经元组来诱导基于使用情况的模块化；（iii）鼓励这些模块的专业化，使其激活与输入类别相关。我们在ImageNet-100和CIFAR-100上评估该方法，结果显示，无需模块级监督即可出现专用模块，这些模块捕获数据中有意义的高级结构，单个模块对语义类别（如狗或交通工具）作出响应。我们还研究了子任务的分层划分如何随模块数量的变化而出现，结果表明，竞争动力学可作为在标准架构中诱导功能模块化的简单机制。
英文摘要
Modularity in deep neural networks has been proposed as a means of improving both interpretability and training by promoting disentangled representations and reducing redundancy. In this work, we study the emergence of modular structure through competition dynamics between groups of neurons during training. We introduce a method that (i) maintains near-baseline accuracy, (ii) induces usage-based modularity by sparsely routing inputs to neuron groups, and (iii) encourages specialization of these modules, such that their activations are correlated with input classes. We evaluate the proposed approach on ImageNet-100 and CIFAR-100 and show that with it, specialized modules emerge without module-level supervision. These modules capture a meaningful high-level structure in the data, with individual modules responding to semantic categories (e.g., dogs or vehicles). We also study the emergence of a hierarchical partition of sub-tasks depending on the number of modules. Our results suggest that competitive dynamics can serve as a simple mechanism for inducing functional modularity in standard architectures.
发表机构
- Univ. Rennes Inria（雷恩大学 Inria 研究所）
- University of Paris 8（巴黎第八大学）
机构由 AI 辅助整理，请以论文原文为准。
