There is an increasing interest in developing LLMs for medical diagnosis to improve diagnosis efficiency. Despite their alluring technological potential, there is no unified and comprehensive evaluation criterion, leading to the inability to evaluate the quality and potential risks of medical LLMs, further hindering the application of LLMs in medical treatment scenarios. Besides, current evaluations heavily rely on labor-intensive interactions with LLMs to obtain diagnostic dialogues and human evaluation on the quality of diagnosis dialogue. To tackle the lack of unified and comprehensive evaluation criterion, we first initially establish an evaluation criterion, termed LLM-specific Mini-CEX to assess the diagnostic capabilities of LLMs effectively, based on original Mini-CEX. To address the labor-intensive interaction problem, we develop a patient simulator to engage in automatic conversations with LLMs, and utilize ChatGPT for evaluating diagnosis dialogues automatically. Experimental results show that the LLM-specific Mini-CEX is adequate and necessary to evaluate medical diagnosis dialogue. Besides, ChatGPT can replace manual evaluation on the metrics of humanistic qualities and provides reproducible and automated comparisons between different LLMs.
Site Reliability Engineers (SREs) play a key role in issue identification and resolution. After an issue is reported, SREs come together in a virtual room (collaboration platform) to triage the issue. While doing so, they leave behind a wealth of information which can be used later for triaging similar issues. However, usability of the conversations offer challenges due to them being i) noisy and ii) unlabelled. This paper presents a novel approach for issue artefact extraction from the noisy conversations with minimal labelled data. We propose a combination of unsupervised and supervised mode…
The subject of conversational mining has become of great interest recently due to the explosion of social and other online media. Supplementing this explosion of text is the advancement in pre-trained language models which have helped us to leverage these sources of information. An interesting domain to analyse is conversations in terms of complexity and value. Complexity arises due to the fact that a conversation can be asynchronous and can involve multiple parties. It is also computationally intensive to process. We use unsupervised methods in our work in order to develop a conversational pa…
Personal knowledge bases (PKBs) are critical to many applications, such as Web-based chatbots and personalized recommendation. Conversations containing rich personal knowledge can be regarded as a main source to populate the PKB. Given a user, a user attribute, and user utterances from a conversational system, we aim to predict the personal attribute value for the user, which is helpful for the enrichment of PKBs. However, there are three issues existing in previous studies: (1) manually labeled utterances are required for model training; (2) personal attribute knowledge embedded in both utter…
Chit-Chat or Deep Talk: Prompt Engineering for Process Mining
Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（RUDN大学））；Joint Institute for Nuclear Research（联合核研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（舍夫德大学）
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
Institute for Logic, Language and Computation, University of Amsterdam（逻辑、语言与计算研究所，阿姆斯特丹大学）；Khoury College of Computer Sciences, Northeastern University（计算机科学学院，东北大学）；Center for Language and Cognition, University of Groningen（语言与认知中心，格罗宁根大学）
A Computational Approach to Modeling Conversational Systems: Analyzing Large-Scale Quasi-Patterned Dialogue Flows
Software Engineering and Mathematics Department（软件工程与数学系）；National Institute of Applied Science and Technology（应用科学和技术国家研究院）；Computer Science Department（计算机科学系）；University of Tunis El Manar（突尼斯艾尔曼纳大学）
Towards Conversational AI for Human-Machine Collaborative MLOps
Innov-Acts Ltd（Innov-Acts公司）；Dept. of Digital Systems（数字系统系）；University of Piraeus（比雷埃克斯大学）；Dept. of Informatics and Telematics（信息与电信学系）；Harokopio University（哈罗科比奥大学）
Conversational Complexity for Assessing Risk in Large Language Models
University of Cambridge（剑桥大学）；Spanish National Research Council（西班牙国家研究委员会）；Universitat Politècnica de València（瓦伦西亚理工大学）；Leverhulme Centre for the Future of Intelligence（勒沃休姆未来智能中心）；Center for Automation and Robotics（自动化与机器人中心）；Valencian Research Institute for Artificial Intelligence（瓦伦西亚人工智能研究所）
Context-Aware Classification and Grading of Sensitive Information in Online Conversational Health Data
School of Computing, Macquarie University, Australia（麦考瑞大学计算机学院）；Department of Biomedical Engineering, National University of Singapore（新加坡国立大学生物医学工程系）；School of Mathematics and Statistics, Shandong University of Technology, China（山东理工大学数学与统计学院）；Digital Intelligence Center, Fuzhou University Affiliated Provincial Hospital, China（福州市附属省医院数字智能中心）；School of Computer Science and Engineering, UNSW, Australia（新南威尔士大学计算机科学与工程学院）
Detecting Conversational Mental Manipulation with Intent-Aware Prompting
The University of Sydney（悉尼大学）；University of Technology Sydney（悉尼科技大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Harvard University（哈佛大学）；University of New South Wales（新南威尔士大学）
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Changi General Hospital（樟宜综合医院）
Data for human-human spoken dialogues for research and development are currently very limited in quantity, variety, and sources; such data are even scarcer in healthcare. In this work, we investigate fast prototyping of a dialogue comprehension system by leveraging on minimal nurse-to-patient conversations. We propose a framework inspired by nurse-initiated clinical symptom monitoring conversations to construct a simulated human-human dialogue dataset, embodying linguistic characteristics of spoken interactions like thinking aloud, self-contradiction, and topic drift. We then adopt an establis…
Vienna University of Economics and Business（维也纳经济大学）；Federal University of Rio de Janeiro（里约热内卢联邦大学）；University of Amsterdam（阿姆斯特丹大学）
Understanding the structure of interaction processes helps us to improve information-seeking dialogue systems. Analyzing an interaction process boils down to discovering patterns in sequences of alternating utterances exchanged between a user and an agent. Process mining techniques have been successfully applied to analyze structured event logs, discovering the underlying process models or evaluating whether the observed behavior is in conformance with the known process. In this paper, we apply process mining techniques to discover patterns in conversational transcripts and extract a new model…
Knowledge Graph(KG) grounded conversations often use large pre-trained models and usually suffer from fact hallucination. Frequently entities with no references in knowledge sources and conversation history are introduced into responses, thus hindering the flow of the conversation -- existing work attempt to overcome this issue by tweaking the training procedure or using a multi-step refining method. However, minimal effort is put into constructing an entity-level hallucination detection system, which would provide fine-grained signals that control fallacious content while generating responses…
Avaya（Avaya公司）；Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）；Poznan University of Technology（波兹南理工大学）；AGH University of Science and Technology（AGH科技大学）
Avaya Conversational Intelligence(ACI) is an end-to-end, cloud-based solution for real-time Spoken Language Understanding for call centers. It combines large vocabulary, real-time speech recognition, transcript refinement, and entity and intent recognition in order to convert live audio into a rich, actionable stream of structured events. These events can be further leveraged with a business rules engine, thus serving as a foundation for real-time supervision and assistance applications. After the ingestion, calls are enriched with unsupervised keyword extraction, abstractive summarization, an…
Dialogue level quality estimation is vital for optimizing data driven dialogue management. Current automated methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce the generalizability of the trained models. We propose a novel user satisfaction estimation approach which minimizes an adaptive multi-task loss function in order to jointly predict turn-level Response Quality labels provided by experts and explicit dialogue-level ratings provided by end users. The proposed BiLSTM based deep neural net model aut…
Division of Science and Technology, Beijing Normal University-Hong Kong Baptist University United International College（北京师范大学-香港浸会大学联合国际学院科技学部）
In this paper, we proposed a transfer learning-based English language learning chatbot, whose output generated by GPT-2 can be explained by corresponding ontology graph rooted by fine-tuning dataset. We design three levels for systematically English learning, including phonetics level for speech recognition and pronunciation correction, semantic level for specific domain conversation, and the simulation of free-style conversation in English - the highest level of language chatbot communication as free-style conversation agent. For academic contribution, we implement the ontology graph to expla…
