EventChat: 一种基于大语言模型的对话推荐系统在中小企业休闲活动探索中的实现与用户导向评估
EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context
AI总结：
本文提出一种基于大语言模型的对话推荐系统，通过客观指标和用户评价评估其在中小企业场景下的性能，揭示了系统在用户体验和商业可行性方面的挑战。
摘要：
大语言模型（LLMs）在对话推荐系统（CRS）的战略潜力上带来了巨大变革。然而，迄今为止，研究主要集中在技术框架上，而非终端用户评估或企业战略影响，尤其是对构成全球经济基础的中小企业（SME）而言。本文详细描述了在SME环境中设计的LLM驱动的CRS及其在实际应用中的表现，结合客观系统指标和主观用户评估。同时，我们还提出了一个简化的修订ResQue模型来评估LLM驱动的CRS，以在快速发展的领域中实现可复制性。我们的结果表明，从用户体验角度看系统性能良好（85.5%的推荐准确性），但突显了延迟、成本和质量问题对商业可行性的挑战。值得注意的是，每次交互的中位成本为0.04美元，延迟为5.7秒，成本效益和响应时间成为实现更用户友好且经济可行的LLM驱动CRS的关键因素。这些成本的主要驱动因素是将先进LLM用作检索增强生成（RAG）技术中的排序器。此外，我们的结果还表明，仅依赖诸如基于提示的学习方法（使用ChatGPT作为底层LLM）在生产环境中难以实现令人满意的质量。本文还概述了中小企业部署LLM驱动CRS的战略考虑，特别是在当前技术景观中的权衡。
英文摘要：
Large language models (LLMs) present an enormous evolution in the strategic potential of conversational recommender systems (CRS). Yet to date, research has predominantly focused upon technical frameworks to implement LLM-driven CRS, rather than end-user evaluations or strategic implications for firms, particularly from the perspective of a small to medium enterprises (SME) that makeup the bedrock of the global economy. In the current paper, we detail the design of an LLM-driven CRS in an SME setting, and its subsequent performance in the field using both objective system metrics and subjective user evaluations. While doing so, we additionally outline a short-form revised ResQue model for evaluating LLM-driven CRS, enabling replicability in a rapidly evolving field. Our results reveal good system performance from a user experience perspective (85.5% recommendation accuracy) but underscore latency, cost, and quality issues challenging business viability. Notably, with a median cost of $0.04 per interaction and a latency of 5.7s, cost-effectiveness and response time emerge as crucial areas for achieving a more user-friendly and economically viable LLM-driven CRS for SME settings. One major driver of these costs is the use of an advanced LLM as a ranker within the retrieval-augmented generation (RAG) technique. Our results additionally indicate that relying solely on approaches such as Prompt-based learning with ChatGPT as the underlying LLM makes it challenging to achieve satisfying quality in a production environment. Strategic considerations for SMEs deploying an LLM-driven CRS are outlined, particularly considering trade-offs in the current technical landscape.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
