Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Bias

Strong inductive biases provably prevent harmless interpolation

Classical wisdom suggests that estimators should avoid fitting noise to achieve good generalization. In contrast, modern overparameterized models can yield small test error despite interpolating noise -- a phenomenon often called "benign overfitting" or "harmless interpolation". This paper argues that the degree to which interpolation is harmless hinges upon the strength of an estimator's inductive bias, i.e., how heavily the estimator favors solutions with a certain structure: while strong inductive biases prevent harmless interpolation, weak inductive biases can even require fitting noise to…

谁是（多）公平之最：从多校准视角重新思考基于插值的数据增强

Who's the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration

针对现有插值数据增强方法公平性评估未考虑不确定性和多少数群体的问题，本文用多校准检验Fair Mixup等方法，发现原始Mixup更优，结合多校准后处理可进一步提升公平性，解决了少数群体数据稀疏时的多校准改进难题。

插值MLP：可控的归纳偏置

Interpolated-MLPs: Controllable Inductive Bias

提出插值MLP（I-MLP）方法，通过在先验模型固定权重间进行插值来实现对归纳偏置的分数级控制，实验发现在低算力视觉任务中归纳偏置与性能呈连续双向对数关系。

Learning the Travelling Salesperson Problem Requires Rethinking Generalization

End-to-end training of neural network solvers for graph combinatorial optimization problems such as the Travelling Salesperson Problem (TSP) have seen a surge of interest recently, but remain intractable and inefficient beyond graphs with few hundreds of nodes. While state-of-the-art learning-driven approaches for TSP perform closely to classical solvers when trained on trivially small sizes, they are unable to generalize the learnt policy to larger instances at practical scales. This work presents an end-to-end neural combinatorial optimization pipeline that unifies several recent papers in o…

A law of adversarial risk, interpolation, and label noise

In supervised learning, it has been shown that label noise in the data can be interpolated without penalties on test accuracy. We show that interpolating label noise induces adversarial vulnerability, and prove the first theorem showing the relationship between label noise and adversarial risk for any data distribution. Our results are almost tight if we do not make any assumptions on the inductive bias of the learning algorithm. We then investigate how different components of this problem affect this result, including properties of the distribution. We also discuss non-uniform label noise dis…

算法任务捕获、计算复杂性与无限变换器的归纳偏差

Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers

研究定义了变换器在任意任务规模下可控误差和对数样本适应的能力，分析了无限宽度变换器在不同 regime 中的计算复杂性上限，并揭示了变换器在高效多项式时间启发式方案中对高复杂度算法的偏倚。