We study Rademacher processes where the coefficients are functions evaluated at fixed, but arbitrary covariables. Specifically, we assume the function class under consideration to be parametrized by the standard cocube in l dimensions and we are mainly interested in the high-dimensional, asymptotic situation, that is, l as well the number of Rademacher variables n go to infinity with l much larger than n. We refine and apply classical entropy bounds and Majorizing Measures, both going back to the well known idea of chaining. That way, we derive general upper bounds for Rademacher processes. In the linear case and under high correlations, we further improve on these bounds. In particular, we give bounds independent of l for highly correlated covariables.
Lean Formalization of Generalization Error Bound by Rademacher Complexity and Dudley's Entropy Integral
RIKEN AIP(日本理化学研究所AIP);CyberAgent Inc.(CyberAgent公司);OMRON SINIC X Corporation(OMRON SINIC X株式会社);University College Cork(科克大学);The University of Tokyo(东京大学)
Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards
Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算高精尖创新中心);Advanced Institute of Information Technology, Peking University(北京大学信息技术高等研究院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究所);Computer and Mathematical Sciences, Computer Science, and Statistics, University of Toronto(多伦多大学计算机与数学科学、计算机科学和统计学系);MBZUAI(穆罕默德·本·扎耶德人工智能大学)