广义熵正则化或：标签平滑没有什么特别之处
Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing
摘要：
先前的工作已经探索了直接正则化概率模型的输出分布，以减轻峰值（即过度自信）预测，这是过度拟合的常见迹象。这类技术，标签平滑是其中之一，与熵正则化有关。尽管标签平滑在语言生成任务中跨架构和数据集的一致成功，但仍存在两个问题：（1）对熵正则化对模型的潜在影响了解甚少，以及（2）熵正则化技术的整个空间在很大程度上未被探索。我们引入了一个参数族的熵正则化，其中包括标签平滑作为一个特例，并使用它来更好地理解模型的熵和它的性能之间的关系语言生成任务。我们还发现，模型性能的方差可以在很大程度上解释由模型的熵。最后，我们发现，标签平滑证明不允许稀疏的输出分布，一个不可取的属性语言生成模型，因此建议使用其他熵正则化方法在其位置。
英文摘要：
Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i.e. over-confident) predictions, a common sign of overfitting. This class of techniques, of which label smoothing is one, has a connection to entropy regularization. Despite the consistent success of label smoothing across architectures and data sets in language generation tasks, two problems remain open: (1) there is little understanding of the underlying effects entropy regularizers have on models, and (2) the full space of entropy regularization techniques is largely unexplored. We introduce a parametric family of entropy regularizers, which includes label smoothing as a special case, and use it to gain a better understanding of the relationship between the entropy of a model and its performance on language generation tasks. We also find that variance in model performance can be explained largely by the resulting entropy of the model. Lastly, we find that label smoothing provably does not allow for sparsity in an output distribution, an undesirable property for language generation models, and therefore advise the use of other entropy regularization methods in its place.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
