Deep structured-prediction energy-based models combine the expressive power of learned representations and the ability of embedding knowledge about the task at hand into the system. A common way to learn parameters of such models consists in a multistage procedure where different combinations of components are trained at different stages. The joint end-to-end training of the whole system is then done as the last fine-tuning stage. This multistage approach is time-consuming and cumbersome as it requires multiple runs until convergence and multiple rounds of hyperparameter tuning. From this point of view, it is beneficial to start the joint training procedure from the beginning. However, such approaches often unexpectedly fail and deliver results worse than the multistage ones. In this paper, we hypothesize that one reason for joint training of deep energy-based models to fail is the incorrect relative normalization of different components in the energy function. We propose online and offline scaling algorithms that fix the joint training and demonstrate their efficacy on three different tasks.
Structured Scaling of AI Discovery Across Diverse Scientific Domains
Wizard Intelligence Learning Lab, Stanford University(智能巫师学习实验室,斯坦福大学);Peking University(北京大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
The choice of scaling technique matters for classification performance
Universidade Federal de Pernambuco(伯南布哥联邦大学);Universidade Federal de Alagoas(阿拉戈斯联邦大学);École de Technologie Supérieure, Université du Québec(魁北克大学高等技术学院)
Natural Building Blocks for Structured World Models: Theory, Evidence, and Scaling
VERSES AI Research Lab(VERSES AI研究实验室);University of Tübingen(图宾根大学);ELLIS Institute, Tübingen(图宾根ELLIS研究所);MPI for Intelligent Systems, Tübingen(图宾根智能系统研究所)