arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.27652cs.CLcs.AI
收藏

语料库质量提升,但检索效用未提升:语音到问答流水线中预处理与精调的消融研究

Corpus Quality Improves, Retrieval Utility Does Not: An Ablation of Preprocessing and Refinement in a Speech-to-QA Pipeline

Flitto(飞乐拓)

Wonsup Shin, Jingu Kim

AI总结:

本文通过消融研究发现,语音转问答流水线的预处理与精调阶段可提升语料库质量,但无法提升下游检索准确率,且预处理会增加成本,语料库本身对检索准确率有显著提升。

摘要:

使用语言模型流水线构建训练语料的从业者,会保留预处理或精调(后处理)阶段,因为启用这些阶段会提升语料库质量。我们对将会议语音转换为问答对的生产流水线进行消融研究:预处理阶段重写转录文本,精调阶段过滤生成的问答对。通过开启和关闭这两个阶段,从相同的40个韩国医疗和金融会议中得到四个语料库。就语料库本身而言,质量随阶段启用而提升,四个LLM评判者和六位领域人类专家对四个语料库的排序一致,完全精调的语料库在1-5分制下分别领先0.18和0.22。随后我们探究该增益是否会传递到下游:将每个语料库提供给六个模型进行推理并测量检索准确率,排序并未保持;启用转录重写会使检索准确率降低0.061,这是唯一在多重比较校正后仍存在的阶段效应,且该阶段会读取整个会议,使单会议成本几乎翻倍;两个阶段都启用时,准确率无变化,我们预先声明的等价性测试证实了这一点。语料库本身显然有价值,在我们测试的每个模型和领域中,它将检索准确率从0.231提升至0.748,超过三分之一的问答对包含两个前沿模型单独无法生成的知识。在我们的数据中,原假设并非来自弱语料库,这类评分标准的分数提升无法预测语料库对模型的作用。

英文摘要:

Practitioners building training corpora with a pipeline of language models keep a preprocessing or refinement (post-process) stage when switching it on raises the corpus quality. We ablate a production pipeline that turns conference speech into question-answer pairs. The preprocessing stage rewrites the transcript, and the refinement stage filters the generated pairs. Switching each on and off gives four corpora from the same 40 Korean medical and finance sessions. On the corpus itself, quality rises as the stages switch on. Four LLM judges and six human domain experts rank the four corpora the same way, and the fully refined corpus leads by +0.18 and +0.22 on a 1-5 scale. We then ask whether that gain carries downstream. We supply each corpus to six models at inference and measure retrieval accuracy. The ranking does not hold. Enabling the transcript rewrite lowers retrieval accuracy by 0.061, the only stage effect that survives correction for multiple comparisons. That stage also reads the whole session, so it almost doubles the cost of one. With both stages on, accuracy does not change. An equivalence test we declared in advance confirms this. The corpus itself is clearly valuable. It lifts retrieval accuracy from 0.231 to 0.748 in every model and domain we test. More than a third of its pairs carry knowledge that neither of two frontier models produces on its own. On our data, the null does not come from weak corpora. A rising score on a rubric of this kind did not predict what the corpus holds for a model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑