TikTalk：面向真实世界多模态闲聊的基于视频的对话数据集
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
AI总结：
本文提出基于视频的多模态对话数据集TikTalk，包含38K视频和367K对话，揭示了多模态闲聊的新挑战，并评估发现结合知识图谱的模型表现最佳。
摘要：
为了促进具有多模态上下文的智能且类人聊天机器人的研究，我们引入了一个新的基于视频的多模态对话数据集，称为TikTalk。我们从一个流行的视频分享平台收集了38K个视频，以及用户在视频下方发布的367K条对话。用户基于观看视频获得的多模态体验进行自发对话，这有助于重现真实世界的闲聊语境。与以往的多模态对话数据集相比，TikTalk中更丰富的上下文类型带来了更多样化的对话，但也增加了从复杂多模态信息中捕捉人类兴趣以生成个性化回复的难度。此外，我们的数据集中更频繁地唤起了外部知识。这些事实揭示了多模态对话模型面临的新挑战。我们定量展示了TikTalk的特征，提出了一个基于视频的多模态闲聊任务，并评估了几个对话基线。实验结果表明，结合大语言模型（LLM）的模型可以生成更多样化的回复，而利用知识图谱引入外部知识的模型整体表现最佳。此外，目前没有模型能够很好地解决上述所有挑战。即使是带有视觉扩展的LLM，未来仍有很大的改进空间。我们的数据集可在\url{https://ruc-aimind.github.io/projects/TikTalk/}获取。
英文摘要：
To facilitate the research on intelligent and human-like chatbots with multi-modal context, we introduce a new video-based multi-modal dialogue dataset, called TikTalk. We collect 38K videos from a popular video-sharing platform, along with 367K conversations posted by users beneath them. Users engage in spontaneous conversations based on their multi-modal experiences from watching videos, which helps recreate real-world chitchat context. Compared to previous multi-modal dialogue datasets, the richer context types in TikTalk lead to more diverse conversations, but also increase the difficulty in capturing human interests from intricate multi-modal information to generate personalized responses. Moreover, external knowledge is more frequently evoked in our dataset. These facts reveal new challenges for multi-modal dialogue models. We quantitatively demonstrate the characteristics of TikTalk, propose a video-based multi-modal chitchat task, and evaluate several dialogue baselines. Experimental results indicate that the models incorporating large language models (LLM) can generate more diverse responses, while the model utilizing knowledge graphs to introduce external knowledge performs the best overall. Furthermore, no existing model can solve all the above challenges well. There is still a large room for future improvements, even for LLM with visual extensions. Our dataset is available at \url{https://ruc-aimind.github.io/projects/TikTalk/}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
