arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00775cs.SDeess.AS
收藏

基于分布的嵌入距离测量音频提示遵循度

Measuring Audio Prompt Adherence with Distribution-based Embedding Distances

Sony Computer Science Laboratories(索尼计算机科学实验室)

Maarten Grachten

AI总结:

本文研究能否用基于分布的嵌入距离(如FAD)来通用地测量生成式音乐模型对音频提示的遵循度,提出由嵌入模型、投影、嵌入距离和数据融合组成的简单测量流程,并通过实验验证其对音高和时间偏移扰动具有敏感性。

摘要:

越来越多的生成式音乐模型可以以音频提示为条件,该音频提示作为音乐上下文,模型需要为其创建伴奏(通常还会通过文本提示进一步指定)。对模型输出遵循音频提示程度的评估通常以模型特定或问题特定的方式进行,推测是因为尚未出现通用的音频提示遵循度评估方法。这样一种方法在开发与训练新模型时可能有用,也有助于使不同模型之间的性能具有可比性。在本文中,我们研究常用的基于分布的距离(如 Fréchet Audio Distance,FAD)是否可用于测量音频提示遵循度。我们提出一个基于少量组成部分(嵌入模型、投影、嵌入距离和数据融合方法)的简单流程,并使用基线验证对其进行系统性评估。在后续实验中,我们测试所提出的音频遵循度测量对音高和时间偏移扰动的敏感性。结果表明,即使参考分布和候选分布来自不同的音乐集合,所提出的测量对这些扰动仍然敏感。尽管还需要更多实验来回答一些尚未解决的问题,例如该测量对不影响音频提示遵循度的声学伪影的鲁棒性,但当前结果表明,基于分布的嵌入距离为测量音频提示遵循度提供了一种可行的方法。所提出测量的 Python/PyTorch 实现已作为 GitHub 仓库公开提供。

英文摘要:

An increasing number of generative music models can be conditioned on an audio prompt that serves as musical context for which the model is to create an accompaniment (often further specified using a text prompt). Evaluation of how well model outputs adhere to the audio prompt is often done in a model or problem specific manner, presumably because no generic evaluation method for audio prompt adherence has emerged. Such a method could be useful both in the development and training of new models, and to make performance comparable across models. In this paper we investigate whether commonly used distribution-based distances like Fréchet Audio Distance (FAD), can be used to measure audio prompt adherence. We propose a simple procedure based on a small number of constituents (an embedding model, a projection, an embedding distance, and a data fusion method), that we systematically assess using a baseline validation. In a follow-up experiment we test the sensitivity of the proposed audio adherence measure to pitch and time shift perturbations. The results show that the proposed measure is sensitive to such perturbations, even when the reference and candidate distributions are from different music collections. Although more experimentation is needed to answer unaddressed questions like the robustness of the measure to acoustic artifacts that do not affect the audio prompt adherence, the current results suggest that distribution-based embedding distances provide a viable way of measuring audio prompt adherence. An python/pytorch implementation of the proposed measure is publicly available as a github repository.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑