重新思考一致性建模：合成与下游任务
Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks
摘要：
虽然相干建模已经走了很长的路，在开发新的模型，他们的下游应用程序的评估，他们据称开发在很大程度上被忽视。随着神经方法在机器翻译（MT）、摘要和对话系统等应用中的进步，对这些任务的连贯性评估的需求现在比以往任何时候都更加重要。然而，相干模型通常仅在合成任务上进行评估，这可能不能代表其在下游应用中的性能。为了研究合成任务在下游用例中的代表性，我们在合成句子排序任务上对知名的传统和神经连贯性模型进行了基准测试，并将其与三个下游应用程序的性能进行了对比：MT和摘要的连贯性评估，以及基于检索的对话中的下一个话语预测。我们的研究结果表明，在合成任务和下游应用程序中的模型性能之间存在弱相关性，{激励相干模型的替代训练和评估方法。
英文摘要：
Although coherence modeling has come a long way in developing novel models, their evaluation on downstream applications for which they are purportedly developed has largely been neglected. With the advancements made by neural approaches in applications such as machine translation (MT), summarization and dialog systems, the need for coherence evaluation of these tasks is now more crucial than ever. However, coherence models are typically evaluated only on synthetic tasks, which may not be representative of their performance in downstream applications. To investigate how representative the synthetic tasks are of downstream use cases, we conduct experiments on benchmarking well-known traditional and neural coherence models on synthetic sentence ordering tasks, and contrast this with their performance on three downstream applications: coherence evaluation for MT and summarization, and next utterance prediction in retrieval-based dialog. Our results demonstrate a weak correlation between the model performances in the synthetic tasks and the downstream applications, {motivating alternate training and evaluation methods for coherence models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
