arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1905.04447cs.DScs.LGmath.OCstat.ML
收藏

在当前矩阵乘法时间内求解经验风险最小化问题

Solving Empirical Risk Minimization in the Current Matrix Multiplication Time

University of Washington(华盛顿大学);Microsoft Research Redmond(微软研究院雷德蒙德分院);UT-Austin(得克萨斯大学奥斯汀分校);University of California, Berkeley(加利福尼亚大学伯克利分校);Simons Institute for the Theory of Computing(西蒙斯计算理论研究所)

Yin Tat Lee, Zhao Song, Qiuyi Zhang

摘要:

机器学习和计算机科学中的许多凸问题具有相同的形式:\begin{align*} \min_{x} \sum_{i} f_i( A_i x + b_i), \end{align*}其中$f_i$是$\mathbb{R}^{n_i}$上的凸函数,具有常数$n_i$,$A_i \in \mathbb{R}^{n_i \times d}$,$b_i \in \mathbb{R}^{n_i}$和$\sum_i n_i = n$。该问题是线性规划的推广,包含了经验风险最小化中的许多问题。本文给出了一个按时间运行的算法\begin{align*} O^* ( ( n^ω + n^{2.5 - α/2} + n^{2+ 1/6} ) \log (n / δ) ) \end{align*},其中$ω$是矩阵乘法的指数,$α$是矩阵乘法的对偶指数,$δ$是相对精度。请注意,运行时仅对条件编号或其他数据相关参数具有日志依赖性,这些参数在$δ$中捕获。对于当前绑定的$ω\sim 2.38$ [Vassilevska Williams'12,Le Gall'14]和$α\sim 0.31$ [Le Gall,Urrutia'18],我们的运行时$O^* ( n^ω \log (n / δ))$匹配当前最佳解,以解决密集最小二乘回归问题,这是我们考虑的问题的特殊情况。最近,[Alman'18]证明,所有目前已知的技术不能给出一个更好的$ω$低于$2.168$这是大于我们的$2+1/6$。我们的结果推广了最近的结果解决线性规划在当前矩阵乘法时间[科恩,李,宋'19]到更广泛的一类问题。我们的算法提出了两个概念,这是不同的[科恩,李,宋'19]:$\bullet$我们给出了一个强大的确定性中心路径方法,而前一个是一个随机的中心路径更新的随机稀疏向量的权重。$\bullet$我们提出了一种有效的数据结构,即使当权重更新向量是密集的,也能保持内点方法的中心路径。

英文摘要:

Many convex problems in machine learning and computer science share the same form: \begin{align*} \min_{x} \sum_{i} f_i( A_i x + b_i), \end{align*} where $f_i$ are convex functions on $\mathbb{R}^{n_i}$ with constant $n_i$, $A_i \in \mathbb{R}^{n_i \times d}$, $b_i \in \mathbb{R}^{n_i}$ and $\sum_i n_i = n$. This problem generalizes linear programming and includes many problems in empirical risk minimization. In this paper, we give an algorithm that runs in time \begin{align*} O^* ( ( n^ω + n^{2.5 - α/2} + n^{2+ 1/6} ) \log (n / δ) ) \end{align*} where $ω$ is the exponent of matrix multiplication, $α$ is the dual exponent of matrix multiplication, and $δ$ is the relative accuracy. Note that the runtime has only a log dependence on the condition numbers or other data dependent parameters and these are captured in $δ$. For the current bound $ω\sim 2.38$ [Vassilevska Williams'12, Le Gall'14] and $α\sim 0.31$ [Le Gall, Urrutia'18], our runtime $O^* ( n^ω \log (n / δ))$ matches the current best for solving a dense least squares regression problem, a special case of the problem we consider. Very recently, [Alman'18] proved that all the current known techniques can not give a better $ω$ below $2.168$ which is larger than our $2+1/6$. Our result generalizes the very recent result of solving linear programs in the current matrix multiplication time [Cohen, Lee, Song'19] to a more broad class of problems. Our algorithm proposes two concepts which are different from [Cohen, Lee, Song'19] : $\bullet$ We give a robust deterministic central path method, whereas the previous one is a stochastic central path which updates weights by a random sparse vector. $\bullet$ We propose an efficient data-structure to maintain the central path of interior point methods even when the weights update vector is dense.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑