在线线性回归中的特征启动：稀疏遗憾下界与紧单变量速率
Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and Tight Coordinatewise Rates
浏览论文内容
中文总结 AI 辅助
针对Warmuth和Amid在2023年COLT会议提出的开放问题，研究在线线性回归特征启动规则的稀疏遗憾，证明三种规则的遗憾下界，给出匹配数据秩的紧单变量速率，相关多元前沿仍待解决。
中文摘要 AI 辅助
在高维在线预测中，最优预测器可能仅依赖少量特征，因此遗憾应与稀疏度而非环境维度成比例。特征启动（Feature priming）通过从过往数据估计特征权重，并在重新缩放的设计上重新拟合最小范数预测器来实现这一目标。Warmuth和Amid在2023年COLT会议上提出，这类规则中的三种是否有任何一种能提供具有竞争力的在线遗憾保证。我们仅使用基于过往数据的自然Moore–Penrose协议，对该COLT开放问题的稀疏对数形式给出否定答案。我们的分析确定了一个共同阻碍：廉价的干扰插值会导致重新拟合对真正具有预测性的坐标赋予过低权重。精确的目标质量恒等式和双符号论证将此效应转化为截断预测损失。Hadamard构造针对零损失单稀疏比较器，对所有三种规则强制产生Ω(min{T,√d})的遗憾，并扩展至固定素数幂及规则间选择器。相反，遗憾由数据秩控制，欧几里得归一化三角构造匹配该依赖关系，适用于幂次单变量启动，即使在非负第二阶段岭正则化下亦是如此；配对岭构造也覆盖所有三种幂次规则。对冻结语言模型激活的探索性诊断显示出干扰插值、目标权重与损失之间的相同关系。精确的多元和Pearson前沿仍待解决。
英文摘要
In high-dimensional online prediction, sparse comparators motivate regret bounds that depend on sparsity rather than ambient dimension. Feature priming seeks such adaptation by reweighting features using past data and refitting a minimum-norm predictor. At COLT 2023, Warmuth and Amid posed the open problem of whether the univariate, Pearson, or multivariate priming rules admit competitive online regret guarantees. Under the natural past-only Moore--Penrose protocol, we establish sparse-regret lower bounds that refute the corresponding sparse-logarithmic guarantee. The key obstruction is cheap nuisance interpolation, which permits exact interpolation of the history while assigning insufficient weight to the truly predictive coordinate. An exact target-mass identity and a two-sign argument convert this obstruction into clipped prediction loss. Hadamard constructions yield $Ω(\min\{T,\sqrt d\})$ clipped regret for each of the three unit-power rules against a zero-loss one-sparse comparator. For every fixed power $α\ge1$, one shared paired construction further yields linear regret simultaneously for all three powered rules and selectors among them in sufficiently high dimension. A rank upper bound is tight for powered univariate priming, even with Euclidean-unit inputs, and for unit-power Pearson priming with coordinatewise bounded inputs and target-preserving totalization. A separate algebraic construction gives $Ω(\min\{T,d^{1/4}\})$ regret for unit-power multivariate priming under Euclidean-unit inputs. The univariate lower bound persists under any nonnegative second-stage ridge schedule, while a paired ridge construction yields linear lower bounds for all three powered rules. Exploratory diagnostics on frozen language-model activations are consistent with the same qualitative mechanism. The exact multivariate frontier remains open.
发表机构
- Nanyang Technological University（南洋理工大学）
机构由 AI 辅助整理，请以论文原文为准。
