Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.
5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control
University of Information Technology, Ho Chi Minh City, Vietnam（信息技术大学，胡志明市，越南）；Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam（越南胡志明市国家大学，胡志明市，越南）
SpectraQuery: A Hybrid Retrieval-Augmented Conversational Assistant for Battery Science
Mechanical Engineering, Stanford University（斯坦福大学机械工程系）；Applied Energy Division, SLAC National Accelerator Laboratory（SLAC国家加速器实验室应用能源部门）；Institute of Computational and Mathematical Engineering, Stanford University（斯坦福大学计算与数学工程研究所）
The largest store of continually updating knowledge on our planet can be accessed via internet search. In this work we study giving access to this information to conversational agents. Large language models, even though they store an impressive amount of knowledge within their weights, are known to hallucinate facts when generating dialogue (Shuster et al., 2021); moreover, those facts are frozen in time at the point of model training. In contrast, we propose an approach that learns to generate an internet search query based on the context, and then conditions on the search results to finally …
Towards Conversational AI for Human-Machine Collaborative MLOps
Innov-Acts Ltd（Innov-Acts公司）；Dept. of Digital Systems（数字系统系）；University of Piraeus（比雷埃克斯大学）；Dept. of Informatics and Telematics（信息与电信学系）；Harokopio University（哈罗科比奥大学）
MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Meituan Inc.（美团公司）；NiuTrans Research（牛译研所）；Tsinghua University（清华大学）
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Huawei Poisson Lab（华为泊松实验室）；Waseda University（早稻田大学）
Large Language Model-Powered Conversational Agent Delivering Problem-Solving Therapy (PST) for Family Caregivers: Enhancing Empathy and Therapeutic Alliance Using In-Context Learning
Florida State University（佛罗里达州立大学）；University of Washington（华盛顿大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Towards Explainable Conversational AI for Early Diagnosis with Large Language Models
Department of Information and Communication Technology（信息与通信技术系）；Bangladesh University of Professionals（孟加拉专业大学）；Institute of Information Technology（信息技术研究所）；Jahangirnagar University（贾汗吉尔纳瓦德大学）
This paper summarizes our work on the first track of the ninth Dialog System Technology Challenge (DSTC 9), "Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access". The goal of the task is to generate responses to user turns in a task-oriented dialog that require knowledge from unstructured documents. The task is divided into three subtasks: detection, selection and generation. In order to be compute efficient, we formulate the selection problem in terms of hierarchical classification steps. We achieve our best results with this model. Alternatively, we e…
