Re$^3$Dial：用于长轮次开放域对话预训练的对话语料库检索、重组与重缩放
Re$^3$Dial: Retrieve, Reorganize and Rescale Dialogue Corpus for Long-Turn Open-Domain Dialogue Pre-training
AI总结：
针对预训练语料中长轮次对话稀缺导致模型长程上下文利用受限的问题，提出Re$^3$Dial框架，通过检索、重组和重缩放短轮次对话自动构建十亿规模长轮次对话语料库，显著提升了对话模型生成合理且信息丰富回复的能力。
摘要：
在大规模开放域对话数据上进行预训练可以显著提升对话模型的性能。然而，由于长轮次对话会话的稀缺性，预训练对话模型利用长程上下文的能力受到限制。现有预训练语料库中的大多数对话包含的对话轮次少于三轮。为了缓解这一问题，我们提出了Retrieve, Reorganize and Rescale框架（Re$^3$Dial），该框架能够通过重组现有的短轮次对话，自动构建十亿规模的长轮次对话。给定一个短轮次会话，Re$^3$Dial首先使用会话检索器检索连贯的连续会话。为此，我们通过对比训练来训练检索器，以捕获多轮对话内的语义和话语关系。接下来，Re$^3$Dial遵循多样性采样策略从检索结果中采样会话，该策略旨在惩罚重复或通用的会话。然后通过拼接原始会话和采样的会话来生成更长的会话。通过重复上述过程，Re$^3$Dial可以生成连贯的长轮次对话。在多个多轮对话基准上的广泛实验表明，Re$^3$Dial显著提升了对话模型利用长程上下文的能力，从而生成更合理且信息更丰富的回复。最后，我们构建了一个使用Re$^3$Dial高效重缩放对话的工具包，使我们能够构建一个包含1B（十亿）条中文对话会话的语料库，平均轮次为11.3轮（比原始语料库长5倍）。我们的检索器模型、代码和数据已在\url{https://github.com/thu-coai/Re3Dial}公开发布。
英文摘要：
Pre-training on large-scale open-domain dialogue data can substantially improve the performance of dialogue models. However, the pre-trained dialogue model's ability to utilize long-range context is limited due to the scarcity of long-turn dialogue sessions. Most dialogues in existing pre-training corpora contain fewer than three turns of dialogue. To alleviate this issue, we propose the Retrieve, Reorganize and Rescale framework (Re$^3$Dial), which can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. Given a short-turn session, Re$^3$Dial first employs a session retriever to retrieve coherent consecutive sessions. To this end, we train the retriever to capture semantic and discourse relations within multi-turn dialogues through contrastive training. Next, Re$^3$Dial samples a session from retrieved results following a diversity sampling strategy, which is designed to penalize repetitive or generic sessions. A longer session is then derived by concatenating the original session and the sampled session. By repeating the above process, Re$^3$Dial can yield a coherent long-turn dialogue. Extensive experiments on multiple multi-turn dialogue benchmarks demonstrate that Re$^3$Dial significantly improves the dialogue model's ability to utilize long-range context and thus generate more sensible and informative responses. Finally, we build a toolkit for efficiently rescaling conversations with Re$^3$Dial, which enables us to construct a corpus containing 1B Chinese dialogue sessions with 11.3 turns on average (5$\times$ longer than the original corpus). Our retriever model, code, and data is publicly available at \url{https://github.com/thu-coai/Re3Dial}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
