arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07696cs.CL
收藏

用于语音表征的可持续自监督学习

Sustainable self-supervised learning for speech representations

Orange(Orange公司)

Luis Lugo, Valentin Vielzeuf

AI总结:

针对语音自监督模型高能耗问题,提出结合神经层与训练优化的可持续模型,在单GPU上不到一天完成预训练,降低内存与计算成本并改善下游错误率。

摘要:

可持续人工智能聚焦于数据、硬件和算法,以使机器学习模型在环境方面更负责任。特别是,用于语音表征的机器学习模型计算开销巨大,并因其高能耗而引发环境担忧。因此,我们提出一种可持续的自监督模型来学习语音表征,该模型结合了神经层和训练方面的优化,以降低计算成本。所提出的模型在一个资源高效的基线之上进行了改进,同时降低了内存使用和计算成本估算。它使用单块GPU进行预训练,耗时不到一天。除此之外,在下游任务评估中,它还改善了基线的错误率性能。将其与大型语音表征方法相比,其内存使用降低了一个数量级,而计算成本的降低则体现为近三个数量级的改善。

英文摘要:

Sustainable artificial intelligence focuses on data, hardware, and algorithms to make machine learning models more environmentally responsible. In particular, machine learning models for speech representations are computationally expensive, generating environmental concerns because of their high energy consumption. Thus, we propose a sustainable self-supervised model to learn speech representation, combining optimizations in neural layers and training to reduce computing costs. The proposed model improves over a resource-efficient baseline, reducing both memory usage and computing cost estimations. It pretrains using a single GPU in less than a day. On top of that, it improves the error rate performance of the baseline in downstream task evaluations. When comparing it to large speech representation approaches, there is an order of magnitude reduction in memory usage, while computing cost reductions represent almost three orders of magnitude improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑