arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.15145cs.LGcs.DS
收藏

并行化 Boosting 的代价

The Cost of Parallelizing Boosting

UC Berkeley(加州大学伯克利分校);Tsinghua University(清华大学)

Xin Lyu, Hongxun Wu, Junzhao Yang

AI总结:

本文研究 Boosting 算法并行化的代价,证明任何 Boosting 算法要么需要 Ω(1/γ²) 轮交互,要么承受 exp(d) 的复杂度膨胀,并给出达到该下界的算法,首次刻画并行性与总工作量之间的权衡。

摘要:

我们延续 Karbasi 和 Larsen 最近的工作,研究用于学习的弱到强 Boosting 算法并行化的代价。我们的主要结果有两方面:首先,我们证明了一个紧的下界,表明即使对 Boosting 进行“轻微”并行化,也会导致训练复杂度的指数级膨胀。具体而言,设 γ 为弱学习器相对于随机猜测的优势。著名的 AdaBoost 算法通过与弱学习器交互约 Õ(1/γ²) 轮来产生准确假设,其中每一轮都在多项式时间内运行。Karbasi 和 Larsen 证明了“显著”并行化必然导致指数级膨胀:任何 Boosting 算法要么与弱学习器交互 Ω(1/γ) 轮,要么在训练复杂度上产生 exp(d/γ) 的膨胀,其中 d 是假设类的 VC 维。我们通过证明任何 Boosting 算法要么具有 Ω(1/γ²) 轮交互,要么产生更小的 exp(d) 指数级膨胀,从而填补了这一空白。作为对下界的补充,我们证明存在一种 Boosting 算法,使用 Õ(1/(tγ²)) 轮交互,并且仅承受 exp(d·t²) 的膨胀。代入 t = ω(1),这表明我们下界中更小的膨胀是紧的。更有趣的是,这首次给出了 Boosting 中并行性与所需总工作量之间的权衡。

英文摘要:

We study the cost of parallelizing weak-to-strong boosting algorithms for learning, following the recent work of Karbasi and Larsen. Our main results are two-fold: - First, we prove a tight lower bound, showing that even "slight" parallelization of boosting requires an exponential blow-up in the complexity of training. Specifically, let $γ$ be the weak learner's advantage over random guessing. The famous \textsc{AdaBoost} algorithm produces an accurate hypothesis by interacting with the weak learner for $\tilde{O}(1 / γ^2)$ rounds where each round runs in polynomial time. Karbasi and Larsen showed that "significant" parallelization must incur exponential blow-up: Any boosting algorithm either interacts with the weak learner for $Ω(1 / γ)$ rounds or incurs an $\exp(d / γ)$ blow-up in the complexity of training, where $d$ is the VC dimension of the hypothesis class. We close the gap by showing that any boosting algorithm either has $Ω(1 / γ^2)$ rounds of interaction or incurs a smaller exponential blow-up of $\exp(d)$. -Complementing our lower bound, we show that there exists a boosting algorithm using $\tilde{O}(1/(t γ^2))$ rounds, and only suffer a blow-up of $\exp(d \cdot t^2)$. Plugging in $t = ω(1)$, this shows that the smaller blow-up in our lower bound is tight. More interestingly, this provides the first trade-off between the parallelism and the total work required for boosting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑