用于模型训练的后见之明日志
Hindsight Logging for Model Training
摘要:
在现代机器学习中,模型训练是一个迭代的实验过程,可能会消耗大量的计算资源和开发人员的时间。为了帮助这个过程,经验丰富的模型开发人员在训练运行期间记录和可视化程序变量。详尽记录所有变量是不可行的。乐观日志记录可以伴随程序检查点;这允许开发人员事后添加日志语句,并从检查点“重放”所需的日志语句-我们称之为事后日志记录的过程。不幸的是,后见之明日志记录在数据管理和软件工程中引起了棘手的问题。如果做得不好,事后日志可能会浪费资源,并产生包含在多种训练代码变体中的技术债务。在本文中,我们提出了高效和有效的模型训练的日志记录实践的方法,重点是后见之明的日志记录技术。我们的目标是让有经验的模型开发人员学习并采用这些实践。为了使这更容易,我们提供了一个开源的快速低开销恢复工具套件(flor),它体现了我们在三个任务中的设计:(i)Python中的高效后台日志记录,(ii)可适应的周期性检查点,以及(iii)一个仪器库,用于编码事后日志记录,以实现模型训练的高效和自动记录重放。模型开发人员可以单独使用每个flor工具,如果他们认为合适的话,或者他们可以在免提模式下使用flor,委托它端到端地检测他们的代码,以实现高效的记录重放。我们的解决方案利用了数据库系统中的生理事务日志和恢复技术。对现代ML基准测试的评估表明,flor可以以较小的用户可指定开销(例如7%)产生快速检查点,并且仍然提供比从头开始训练快几个数量级的事后日志重放时间。
英文摘要:
In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible. Optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint -- a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptable periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。