用于合成流量生成任务的语言生成模型的评估指标
Evaluation Metrics of Language Generation Models for Synthetic Traffic Generation Tasks
AI总结：
本文指出常见NLG指标不适用于合成流量生成（STG）任务，提出多个比较生成流量与真实用户文本分布的新指标，并通过自动验证和人工标注证明其有效性，将人类判断一致性提升20%。
摘要：
许多自然语言生成（NLG）任务旨在给定输入提示时生成单一输出文本。其他设置则需要生成多个文本，例如用于合成流量生成（STG）。此生成任务对于训练和评估问答系统以及对话智能体至关重要，其目标是生成多个问题或话语，以类似于真实用户的语言变异性。在本文中，我们展示了常见的NLG指标（如BLEU）不适合评估STG。我们提出并评估了几个旨在将生成的流量与真实用户文本分布进行比较的指标。我们通过自动程序验证了我们的指标，以验证它们是否捕获了生成数据的不同类型的质量问题；我们还进行了人工标注，以验证与人类判断的相关性。在三个任务（即购物话语生成、产品问题生成和查询自动完成）上的实验表明，我们的指标对评估STG任务有效，并且与常见的NLG指标相比，将人类判断的一致性提高了20%。我们相信这些发现可以为更好地估计合成文本数据的代表性铺平道路。
英文摘要：
Many Natural Language Generation (NLG) tasks aim to generate a single output text given an input prompt. Other settings require the generation of multiple texts, e.g., for Synthetic Traffic Generation (STG). This generation task is crucial for training and evaluating QA systems as well as conversational agents, where the goal is to generate multiple questions or utterances resembling the linguistic variability of real users. In this paper, we show that common NLG metrics, like BLEU, are not suitable for evaluating STG. We propose and evaluate several metrics designed to compare the generated traffic to the distribution of real user texts. We validate our metrics with an automatic procedure to verify whether they capture different types of quality issues of generated data; we also run human annotations to verify the correlation with human judgements. Experiments on three tasks, i.e., Shopping Utterance Generation, Product Question Generation and Query Auto Completion, demonstrate that our metrics are effective for evaluating STG tasks, and improve the agreement with human judgement up to 20% with respect to common NLG metrics. We believe these findings can pave the way towards better solutions for estimating the representativeness of synthetic text data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
