Dialogue assessment plays a critical role in the development of open-domain dialogue systems. Existing work are uncapable of providing an end-to-end and human-epistemic assessment dataset, while they only provide sub-metrics like coherence or the dialogues are conversed between annotators far from real user settings. In this paper, we release a large-scale dialogue quality assessment dataset (DiQAD), for automatically assessing open-domain dialogue quality. Specifically, we (1) establish the assessment criteria based on the dimensions conforming to human judgements on dialogue qualities, and (2) annotate large-scale dialogues that conversed between real users based on these annotation criteria, which contains around 100,000 dialogues. We conduct several experiments and report the performances of the baselines as the benchmark on DiQAD. The dataset is openly accessible at https://github.com/yukunZhao/Dataset_Dialogue_quality_evaluation.
PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment
National University of Singapore（新加坡国立大学）；Universidad Politécnica de Madrid（马德里理工大学）；Robert Bosch (SEA) Pte Ltd（博世（东南亚）私人有限公司）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Achieving Reliable Human Assessment of Open-Domain Dialogue Systems
ADAPT Centre（ADAPT中心）；Dublin City University（都柏林城市大学）；Trinity College Dublin（都柏林圣三一学院）；SFI Centre for Research Training in Machine Learning（爱尔兰科学基金会机器学习研究培训中心）；Huawei（华为）；Noah’s Ark Lab（诺亚方舟实验室）
MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation
National University of Singapore（新加坡国立大学）；Robert Bosch (SEA)（罗伯特·博世（东南亚）公司）；Kriston AI Lab（克里斯顿人工智能实验室）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
Unsupervised Flow Discovery from Task-oriented Dialogues
CISUC, Universidade de Coimbra（科英布拉大学系统与计算机工程中心，科英布拉大学）；DEI, FCTUC, Universidade de Coimbra（科英布拉大学工程学院，科英布拉大学）；ISEC, Instituto Politécnico de Coimbra（科英布拉理工学院高等工程学院）
Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12
INESC-ID；Instituto Superior Técnico - University of Lisbon（理工学院 - 里斯本大学）；Department of Technology, Operations, and Statistics, New York University（技术、运营与统计系，纽约大学）；Speech Technology and Machine Learning Group - Universidad Politécnica de Madrid（语音技术与机器学习小组 - 马德里理工大学）；Carnegie Mellon University（卡内基梅隆大学）
CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges
ZJU–UIUC Institute（浙大UIUC研究院）；ZJU–Hangzhou Global Scientific and Technological Innovation Center（浙大杭州全球科技创新中心）；College of Computer Science and Technology（计算机科学与技术学院）；Alibaba Group（阿里巴巴集团）
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology（SRM科学技术学院）；SRM University, AP（SRM大学（安得拉邦））
A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators
National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China（电子科技大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；University of Liverpool（利物浦大学）；City University of Hong Kong（香港城市大学）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；China Mobile Research Institute（中国移动研究院）；China Mobile (Suzhou) Software Technology Co., Ltd（中国移动苏州软件技术有限公司）
A Proactive Multi-Agent Dialogue Framework for Assessing Social Language Disorder Traits in Autism
Department of Computer Science（计算机科学系）；University at Albany（阿尔巴尼大学）；Department of Management Information System（管理信息系统系）；West Virginia University（西弗吉尼亚大学）；Department of Radiology（放射学系）；Washington University in St. Louis（圣路易斯华盛顿大学）；Humanities and Social Sciences（人文学与社会科学）
A Video-grounded Dialogue Dataset and Metric for Event-driven Activities
National Institute of Advanced Industrial Science and Technology (AIST)（日本国立先进工业科学技术研究所（AIST））；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）；UC Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；University of Science and Technology of China（中国科学技术大学）
PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)（合肥工业大学计算机与信息工程学院）；Beijing Institute of Technology（北京理工大学）；School of Medical Technology, Beijing Institute of Technology（北京理工大学医学技术学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Hefei Zhongjuyuan Intelligent Technology Co., Ltd.（合肥中聚源智能科技有限公司）
SIMAX: A Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation
Department of Biostatistics and Bioinformatics, Duke University School of Medicine（杜克大学医学学院生物统计学与生物信息学系）；Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School（杜克-新加坡国立大学医学科学院AI+医学科学计划）；Centre for Biomedical Data Science, Duke-NUS Medical School（杜克-新加坡国立大学医学学院生物医学数据科学中心）；Department of Statistical Science, Duke University（杜克大学统计科学系）；Leiden University Medical Centre（莱顿大学医学中心）；Department of Mathematics, University of Texas at Austin（德克萨斯大学奥斯汀分校数学系）；Department of Internal Medicine, Yale School of Medicine（耶鲁医学院内科学系）；Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩尔曼医学院生物统计学、流行病学与信息学系）；The Graduate Group in Applied Mathematics and Computational Science, School of Arts and Sciences, University of Pennsylvania（宾夕法尼亚大学艺术与科学学院应用数学与计算科学联合组）；Medstar Health National Center for Human Factors in Healthcare, Washington, DC, USA（Medstar健康国家人因工程中心，华盛顿特区，美国）；Department of Biomedical Informatics, Columbia University（哥伦比亚大学生物医学信息学系）；Cancer Prevention and Control, Duke Cancer Institute, Durham, NC, USA（杜克癌症研究所癌症预防与控制部，达勒姆，北卡罗来纳州，美国）；Department of Population Health Sciences, Duke University School of Medicine（杜克大学医学学院流行病学与公共卫生系）；Division of Rheumatology and Immunology, Duke University School of Medicine（杜克大学医学学院风湿病学与免疫学系）；Pre-hospital and Emergency Research Centre, Health Services Research and Population Health, Duke-NUS Medical School（杜克-新加坡国立大学医学学院院前急救与应急研究中心，健康服务研究与人口健康）；NUS Artificial Intelligence Institute, National University of Singapore（新加坡国立大学人工智能研究所）；Division of Pulmonary, Allergy and Critical Care Medicine, Duke University School of Medicine（杜克大学医学学院呼吸科、过敏科与危重医学系）；Duke Center for Health Informatics, Duke University（杜克大学健康信息学中心）；Duke Clinical Research Institute, Durham, NC, USA（杜克临床研究中心，达勒姆，北卡罗来纳州，美国）
ODUTQA-MDC: A Task for Open-Domain Underspecified Tabular QA with Multi-turn Dialogue-based Clarification
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Institute of Artificial Intelligence and Future Networks, Beijing Normal University（北京师范大学人工智能与未来网络研究院）；Faculty of Arts and Sciences, Beijing Normal University（北京师范大学文理学院）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港 Baptist大学）
SimulBench: Evaluating Language Models with Creative Simulation Tasks
National University of Singapore（新加坡国立大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Waterloo（滑铁卢大学）；University of Illinois, Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Allen Institute for AI（艾伦人工智能研究所）
StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；School of Information and Library Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校信息与图书馆科学学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（知识驱动的人机智能工程研究中心，教育部，中国）；International Center of Future Science, Jilin University（未来科学国际中心，吉林大学）
