arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.02136cs.HC
收藏

用户意图识别与对大语言模型的满意度:一项基于 ChatGPT 的用户研究

User Intent Recognition and Satisfaction with Large Language Models: A User Study with ChatGPT

Anna Bodonhelyi, Efe Bozkir, Shuo Yang, Enkelejda Kasneci, Gjergji Kasneci

AI总结:

本研究基于意图分类法和提示词重构,分析了 GPT-3.5 Turbo 与 GPT-4 Turbo 的意图识别质量与用户满意度,发现 GPT-4 识别常见意图更优但低频意图较弱,且用户对原始提示词的回答满意度更高。

摘要:

LLMs 的快速发展代表了数字交互和内容参与领域一次具有深远影响的范式转变。尽管它们编码了大量人类生成的知识,并在处理多样化数据类型方面表现出色,但往往面临准确响应特定用户意图的挑战,从而导致用户不满。基于细粒度的意图分类法和基于意图的提示词重构,我们分析了 GPT-3.5 Turbo 和 GPT-4 Turbo 模型基于意图的提示词重构的意图识别质量及用户满意度。本研究强调了人机交互的重要性,并强调了采用跨学科方法改进对话式 AI 系统的必要性。我们表明,GPT-4 在识别常见意图方面优于 GPT-3.5,但在识别较少见的意图时往往被 GPT-3.5 超越。此外,每当用户意图被正确识别时,虽然与 GPT-3.5 相比,用户对 GPT-4 基于意图的重构更为满意,但相较于重构后的提示词,他们往往对模型回答其原始提示词更为满意。我们研究收集的数据已在 GitHub 上公开发布,以供进一步研究。

英文摘要:

The rapid evolution of LLMs represents an impactful paradigm shift in digital interaction and content engagement. While they encode vast amounts of human-generated knowledge and excel in processing diverse data types, they often face the challenge of accurately responding to specific user intents, leading to user dissatisfaction. Based on a fine-grained intent taxonomy and intent-based prompt reformulations, we analyze the quality of intent recognition and user satisfaction with answers from intent-based prompt reformulations of GPT-3.5 Turbo and GPT-4 Turbo models. Our study highlights the importance of human-AI interaction and underscores the need for interdisciplinary approaches to improve conversational AI systems. We show that GPT-4 outperforms GPT-3.5 in recognizing common intents but is often outperformed by GPT-3.5 in recognizing less frequent intents. Moreover, whenever the user intent is correctly recognized, while users are more satisfied with the intent-based reformulations of GPT-4 compared to GPT-3.5, they tend to be more satisfied with the models' answers to their original prompts compared to the reformulated ones. The collected data from our study has been made publicly available on GitHub (https://github.com/ConcealedIDentity/UserIntentStudy) for further research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑