User simulators are essential to conversational AI, enabling scalable agent development and evaluation through simulated interactions. While current Large Language Models (LLMs) have advanced user simulation capabilities, we reveal that they struggle to consistently demonstrate goal-oriented behavior across multi-turn conversations--a critical limitation that compromises their reliability in downstream applications. We introduce User Goal State Tracking (UGST), a novel framework that tracks user goal progression throughout conversations. Leveraging UGST, we present a three-stage methodology for developing user simulators that can autonomously track goal progression and reason to generate goal-aligned responses. Moreover, we establish comprehensive evaluation metrics for measuring goal alignment in user simulators, and demonstrate that our approach yields substantial improvements across two benchmarks (MultiWOZ 2.4 and τ-Bench). Our contributions address a critical gap in conversational AI and establish UGST as an essential framework for developing goal-aligned user simulators.
How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation
School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);Key Laboratory of Big Data & Artificial Intelligence in Transportation(Beijing Jiaotong University), Ministry of Education(教育部交通大数据与人工智能重点实验室(北京交通大学))
BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virginia Tech(弗吉尼亚理工大学);Bangladesh University of Engineering and Technology(孟加拉工程与技术大学);BRAC University(BRAC大学);University of Toronto(多伦多大学)
Impact of Decoding Methods on Human Alignment of Conversational LLMs
Birla Institute of Technology and Science, Pilani(比拉理工与科学学院(皮拉尼));National University of Singapore(新加坡国立大学);NUS Center for Trusted Internet and Community(新加坡国立大学可信互联网与社区中心);Department of Communications and New Media, National University of Singapore(新加坡国立大学传播与新媒体系)
本文研究了Beam Search、Top K Sampling和Nucleus Sampling等解码方法对LLM与人类对话对齐的影响,提出了内容、风格和心理测量导向的新衡量标准,发现较少beam数和较低P值对齐效果更好,且任务导向与开放式数据集表现不同。
The complexities of patient-centred conversational artificial intelligence
Verily Health(Verily健康公司);University of Oxford(牛津大学);University of Birmingham(伯明翰大学);University Hospitals Birmingham NHS Foundation Trust(伯明翰大学医院NHS基金会信托);Harvard Medical School(哈佛医学院);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心)
ConvoDrift: A Multi-Turn Conversational Dataset for Modeling Stylistic Tone Evolution
Informatics Institute of Technology, Sri Lanka(斯里兰卡信息技术学院);National Health and Medical Research Council, Australia(澳大利亚国家健康与医学研究理事会);Luxembourg Centre for Systems Biomedicine (LCSB), University of Luxembourg(卢森堡大学系统生物医学中心)
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Kyoto University(京都大学);Hohai University(河海大学);The University of Tokyo(东京大学);University of Science and Technology of China(中国科学技术大学);Hong Kong Polytechnic University(香港理工大学)
Decoding Workload and Agreement From EEG During Spoken Dialogue With Conversational AI
Chair of Neuroadaptive Human--Computer Interaction(神经适应性人机交互教授席位);Brandenburg Technical University Cottbus--Senftenberg(勃兰登堡技术大学库滕堡-森芬根堡分校);Auryal GmbH(Auryal公司);Cottbus, Germany(库滕堡,德国)
PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Soochow University(苏州大学);University of Science and Technology(科技大学)
Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
Towards Stable and Personalised Profiles for Lexical Alignment in Spoken Human-Agent Dialogue
Centre for Language Studies, Centre for Language and Speech Technology, Radboud University,Nijmegen, The Netherlands(语言研究所以及语言与语音技术中心,拉德堡德大学,尼姆egen,荷兰);Behavioural Science Institute, Radboud University, Nijmegen, The Netherlands(行为科学研究所,拉德堡德大学,尼姆egen,荷兰)
Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants
Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔朗根-纽伦堡弗里德里希-亚历山大大学);BMW Group(宝马集团);Technical University of Munich(慕尼黑工业大学);RWTH Aachen University(亚琛工业大学);fortiss GmbH(fortiss有限公司)
Expectation Confirmation Preference Optimization for Multi-Turn Conversational Recommendation Agent
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China(中国人民大学北京校区人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程研究中心,教育部);Huawei Noah’s Ark Lab, China(华为诺亚方舟实验室)