arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.07277cs.CLcs.HCcs.SDeess.AS
收藏

语音与文本转录:在语音摘要任务中人类标注者会受其影响吗?

Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

Carnegie Mellon University(卡内基梅隆大学);ASAPP Inc(ASAPP公司);Mohammed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学)

Roshan Sharma, Suwon Shon, Mark Lindsey, Hira Dhamyal, Rita Singh, Bhiksha Raj

AI总结:

本研究对比语音和文本转录两种模态下人类标注的语音摘要差异,通过多维度评估发现语音摘要事实一致性与信息筛选性更优,转录摘要受识别错误影响,专家摘要更可靠,并公开数据代码推动领域研究。

摘要:

生成式语音摘要的参考摘要需要人工标注,标注方式既可以是听音频录音,也可以是阅读录音的文本转录内容。本文研究了标注者听录音生成的摘要与阅读转录文本生成的摘要是否存在差异。我们采用了现有的内在评估方法,包括人工评估、自动指标、基于大语言模型(LLM)的评估以及一种基于检索的无参考方法。研究发现,基于不同源模态生成的摘要确实存在差异:基于语音的摘要比基于转录文本的摘要事实一致性更强、信息筛选更精准。同时,基于转录文本的摘要会受到源文本识别错误的影响,而专家撰写的摘要信息量更大、可靠性更高。我们公开了所有收集到的数据和分析代码(https://github.com/cmu-mlsp/interview_humanssum),以助力本研究的复现并推动该领域的研究发展。

英文摘要:

Reference summaries for abstractive speech summarization require human annotation, which can be performed by listening to an audio recording or by reading textual transcripts of the recording. In this paper, we examine whether summaries based on annotators listening to the recordings differ from those based on annotators reading transcripts. Using existing intrinsic evaluation based on human evaluation, automatic metrics, LLM-based evaluation, and a retrieval-based reference-free method. We find that summaries are indeed different based on the source modality, and that speech-based summaries are more factually consistent and information-selective than transcript-based summaries. Meanwhile, transcript-based summaries are impacted by recognition errors in the source, and expert-written summaries are more informative and reliable. We make all the collected data and analysis code public(https://github.com/cmu-mlsp/interview_humanssum) to facilitate the reproduction of our work and advance research in this area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑