ReSee：在开放域对话中通过观察细粒度视觉知识进行回应
ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue
AI总结：
针对现有系统受限于数据集和粗粒度视觉知识的问题，提出ReSee框架，通过引入轮次级和实体级细粒度视觉知识，在自动与人工评估中优于多个最先进方法，生成包含真实视觉概念的回复。
摘要：
将视觉知识融入纯文本对话系统已成为模仿人类思考、想象和交流方式的潜在方向。然而，现有的多模态对话系统要么受限于可用数据集的规模和质量，要么受限于视觉知识的粗粒度概念。为了解决这些问题，我们提供了一种构建多模态对话的新范式，以及在此范式下从纯文本对话扩展而来的两个数据集（ReSee-WoW、ReSee-DD）。我们提出将视觉知识显式拆分为更细的粒度（“轮次级”和“实体级”）。为了进一步提升增强视觉信息的准确性和多样性，我们从互联网或大型图像数据集中检索这些信息。为了证明所提供视觉知识的优越性和通用性，我们提出了一个简单但有效的框架ReSee，通过模态拼接将视觉表示添加到普通对话模型中。我们还针对不同的模型配置和视觉知识设置进行了广泛的实验和消融研究。令人鼓舞的实证结果不仅证明了在实体级和轮次级引入视觉知识的有效性，还验证了提出的模型ReSee在自动和人工评估上优于几种最先进的方法。通过利用文本和视觉知识，ReSee能够生成包含真实世界视觉概念的信息丰富的回复。我们的代码可在 https://github.com/ImKeTT/ReSee 获取。
英文摘要：
Incorporating visual knowledge into text-only dialogue systems has become a potential direction to imitate the way humans think, imagine, and communicate. However, existing multimodal dialogue systems are either confined by the scale and quality of available datasets or the coarse concept of visual knowledge. To address these issues, we provide a new paradigm of constructing multimodal dialogues as well as two datasets extended from text-only dialogues under such paradigm (ReSee-WoW, ReSee-DD). We propose to explicitly split the visual knowledge into finer granularity (``turn-level'' and ``entity-level''). To further boost the accuracy and diversity of augmented visual information, we retrieve them from the Internet or a large image dataset. To demonstrate the superiority and universality of the provided visual knowledge, we propose a simple but effective framework ReSee to add visual representation into vanilla dialogue models by modality concatenations. We also conduct extensive experiments and ablations w.r.t. different model configurations and visual knowledge settings. Empirical, encouraging results not only demonstrate the effectiveness of introducing visual knowledge at both entity and turn level but also verify the proposed model ReSee outperforms several state-of-the-art methods on automatic and human evaluations. By leveraging text and vision knowledge, ReSee can produce informative responses with real-world visual concepts. Our code is available at https://github.com/ImKeTT/ReSee.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
