arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.13166cs.HCcs.IR
收藏

使用大语言模型研究对话式后续查询与用户满意度之间的相关性

Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction

Hyunwoo Kim, Yoonseo Choi, Taehyun Yang, Honggu Lee, Chaneon Park, Yongju Lee, Jin Young Kim, Juho Kim

备注：
Accepted to LLM4Eval @ SIGIR 2024 - The First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究通过分析Naver Cue:的对话日志，提出18种后续查询模式分类体系，并构建LLM分类器（73%准确率），发现澄清查询等模式与用户不满意相关，为对话式搜索体验的自动评估提供信号。

摘要：

随着大语言模型（LLMs）的发展，对话式搜索引擎通过支持多轮自然对话来表达用户的搜索意图，从而改变了用户从网络获取信息的方式。用户的自然对话蕴含了丰富但隐性的信号，这些信号反映了用户的搜索意图以及对搜索结果评估，有助于理解用户对系统的体验。然而，关于用户如何以及为何提出后续查询以继续与对话式搜索引擎的对话，以及后续查询如何预示用户满意度，目前研究尚不充分。通过对商业对话式搜索引擎Naver Cue:的实验室用户评估中250个对话轮次进行定性分析，我们提出了一个包含18种用户后续查询模式的分类体系，涵盖两个主要维度：（1）用户继续对话的动机（N=7）和（2）后续查询的行为（N=11）。与现有的查询重构文献相比，我们发现了后续查询背后的一系列新动机和行为，包括询问主观意见或对引擎的回应提供自然语言反馈。为了以可扩展且高效的方式使用我们的分类体系分析对话式搜索日志，我们构建了一个基于LLM的分类器（准确率为73%）。利用该分类器，我们分析了从Cue:的真实使用日志中收集的2,061个对话元组，并考察了我们的分类体系中的对话模式与满意度之间的相关性。我们的初步发现表明，某些信号与不满意相关，例如澄清查询、排除条件和用后续查询替代条件。我们设想，我们的方法能够通过提供满意度信号和现实用户模拟的基础，为对话式搜索体验的自动评估做出贡献。

英文摘要：

With large language models (LLMs), conversational search engines shift how users retrieve information from the web by enabling natural conversations to express their search intents over multiple turns. Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results to understand user experience with the system. However, it is underexplored how and why users ask follow-up queries to continue conversations with conversational search engines and how the follow-up queries signal users' satisfaction. From qualitative analysis of 250 conversational turns from an in-lab user evaluation of Naver Cue:, a commercial conversational search engine, we propose a taxonomy of 18 users' follow-up query patterns from conversational search, comprising two major axes: (1) users' motivations behind continuing conversations (N = 7) and (2) actions of follow-up queries (N = 11). Compared to the existing literature on query reformulations, we uncovered a new set of motivations and actions behind follow-up queries, including asking for subjective opinions or providing natural language feedback on the engine's responses. To analyze conversational search logs with our taxonomy in a scalable and efficient manner, we built an LLM-powered classifier (73% accuracy). With our classifier, we analyzed 2,061 conversational tuples collected from real-world usage logs of Cue: and examined how the conversation patterns from our taxonomy correlates with satisfaction. Our initial findings suggest some signals of dissatisfactions, such as Clarifying Queries, Excluding Condition, and Substituting Condition with follow-up queries. We envision our approach could contribute to automated evaluation of conversation search experience by providing satisfaction signals and grounds for realistic user simulations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑