用张量特征训练网络加速高维函数学习
Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network
- The University of Maryland（马里兰大学）
- The University of Chicago（芝加哥大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究提出一种引入上下文特征的DNN优化方法，结合随机张量分解策略降低存储成本，实现5至40维高维函数的高效学习。
AI中文摘要：
在本研究中，我们提出一种方法，通过深度神经网络（DNN）加速高维函数学习的优化过程。该优化过程将上下文特征引入DNN的第一层，DNN的参数通过标准梯度下降法优化，同时保持输入特征基固定；在DNN参数优化完成后，特征层有机会更新并调整，之后再恢复DNN的优化。特征层包含两类函数：一类可在定义域上以无矩阵方式快速评估（即秩-1特征），另一类更复杂的特征需先通过张量网络（TN）分解策略分解（即张量特征）。特别地，我们研究了使用离散化与分解策略将预训练DNN提炼为TN的特征添加效果；为高效分解由离散化DNN构建的高维函数，我们采用随机张量分解策略，利用随机化可将高维函数分解的存储成本至少降低8个数量级，通过该方法可高效训练5至40维的模型。
英文摘要：
In this work we present a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN). This optimization procedure introduces contextual features into the first layer of a DNN. The parameters of DNN are optimized via standard gradient descent while keeping the input-feature basis fixed. After optimization of the DNN parameters, the feature layer is provided a chance to update and change before DNN optimization resumes. The feature layer has two types of functions: those that can be evaluated quickly in a matrix-free way on the domain (i.e. rank-1 features) and more complex features that must first be decomposed using tensor network (TN) decomposition strategies (tensor features). In particular, we study the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy. To efficiently decompose high-dimensional functions constructed from discretized DNN, we leverage a randomized tensor decomposition strategy. Using randomization, we are able to reduce the storage cost of decomposing high dimensional functions by at least 8 orders of magnitude. Using this approach, we are able to efficiently train models between 5 and 40 dimensions.
