ScaMo：探索自回归运动生成模型中的缩放定律
ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model
AI总结：
本文提出了包含 Motion FSQ-VAE 和文本前缀自回归 transformer 的可扩展运动生成框架 ScaMo，首次在运动生成中验证了测试损失的对数定律及参数与数据的幂律关系，并据此准确预测了 $1e18$ 计算预算下的最优模型配置。
摘要：
缩放定律已在自然语言处理（NLP）和大规模计算机视觉任务等多个领域得到验证；然而，其在运动生成中的应用仍 largely 未被探索。在本文中，我们引入了一个可扩展的运动生成框架，该框架包括运动分词器 Motion FSQ-VAE 和一个文本前缀自回归 transformer。通过全面的实验，我们观察到了该系统的缩放行为。我们首次确认了在运动生成背景下缩放定律的存在。具体而言，我们的结果表明，我们的前缀自回归模型的归一化测试损失相对于计算预算遵循对数定律。此外，我们还确认了非词汇参数、词汇参数和数据令牌相对于计算预算分别存在幂律关系。利用该缩放定律，我们预测了在 $1e18$ 计算预算下的最优 transformer 大小、词汇表大小和数据需求。当使用最优模型大小、词汇表大小和所需数据训练时，系统的测试损失与预测的测试损失精确对齐，从而验证了缩放定律。
英文摘要：
The scaling law has been validated in various domains, such as natural language processing (NLP) and massive computer vision tasks; however, its application to motion generation remains largely unexplored. In this paper, we introduce a scalable motion generation framework that includes the motion tokenizer Motion FSQ-VAE and a text-prefix autoregressive transformer. Through comprehensive experiments, we observe the scaling behavior of this system. For the first time, we confirm the existence of scaling laws within the context of motion generation. Specifically, our results demonstrate that the normalized test loss of our prefix autoregressive models adheres to a logarithmic law in relation to compute budgets. Furthermore, we also confirm the power law between Non-Vocabulary Parameters, Vocabulary Parameters, and Data Tokens with respect to compute budgets respectively. Leveraging the scaling law, we predict the optimal transformer size, vocabulary size, and data requirements for a compute budget of $1e18$. The test loss of the system, when trained with the optimal model size, vocabulary size, and required data, aligns precisely with the predicted test loss, thereby validating the scaling law.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
