Evaluation of a Bi-Directional Methodology for Automated Assessment of Compliance to Continuous Application of Clinical Guidelines, in the Type 2 Diabetes-Management Domain
Ben Gurion University（本·古里安大学）；Rambam Health Care Campus（拉姆巴姆医疗中心）；Technion - Israel Institute of Technology（以色列理工学院）；Clalit Health Care Services（克拉利特医疗服务机构）
Avner Hatsek, Irit Hochberg, Deeb Daoud Naccache, Aya Biderman, Yuval Shahar
We evaluated the DiscovErr system, in which we had previously implemented a new methodology for assessment of compliance to continuous application of evidence-based clinical guidelines, based on a bidirectional search from the guideline objectives to the patient's longitudinal data, and vice versa. We compared the system comments on 1584 transactions regarding the management, over a mean of 5.23 years, of 10 randomly selected Type 2 diabetes patients, to those of two diabetes experts and a senior family practitioner. After providing their own comments, the experts assessed both the correctness (precision) and the importance of each of the DiscovErr system comments. The completeness (recall or coverage) of the system was computed by comparing its comments to those made by the experts. The system made 279 comments. The experts made 181 unique comments. The completeness of the system was 91% compared to comments made by at least two experts, and 98% when compared to comments made by all three. 172 comments were evaluated by the experts for correctness and importance: All 114 medication-related comments, and a random 35% of the 165 monitoring-related comments. The system's correctness was 81% compared to comments judged as correct by both diabetes experts, and 91% compared to comments judged as correct by a diabetes expert and at least as partially correct by the other. 89% of the comments were judged as important by both diabetes experts, 8% were judged as important by one expert, 3% were judged as less important by both experts. The completeness scores of the three experts (compared to the comments of all experts plus the validated system comments) were 75%, 60%, and 55%; the experts' correctness scores (compared to their majority) were respectively 99%, 91%, and 88%. Conclusion: Systems such as DiscovErr can assess the quality of continuous guideline-based care.
Clinicians often do not sufficiently adhere to evidence-based clinical guidelines in a manner sensitive to the context of each patient. It is important to detect such deviations, typically including redundant or missing actions, even when the detection is performed retrospectively, so as to inform both the attending clinician and policy makers. Furthermore, it would be beneficial to detect such deviations in a manner proportional to the level of the deviation, and not to simply use arbitrary cut-off values. In this study, we introduce a new approach for automated guideline-based quality assess…
Université Paris 13（巴黎第十三大学）；Société de Formation Thérapeutique du Généraliste (SFTG)（全科医生治疗培训学会（SFTG））
Well-designed medical decision support system (DSS) have been shown to improve health care quality. However, before they can be used in real clinical situations, these systems must be extensively tested, to ensure that they conform to the clinical guidelines (CG) on which they are based. Existing methods cannot be used for the systematic testing of all possible test cases. We describe here a new exhaustive dynamic verification method. In this method, the DSS is considered to be a black box, and the Quinlan C4.5 algorithm is used to build a decision tree from an exhaustive set of DSS input vect…
Université Paris 13（巴黎第十三大学）；AP-HP（巴黎公立医院集团）；INSERM（法国国家健康与医学研究院）；Silk Informatique（Silk Informatique公司）；Université Paris Descartes（巴黎笛卡尔大学）；Société de Formation Thérapeutique du Généraliste（全科医生治疗培训协会）
Clinical decision support systems have been developed to help physicians to take clinical guidelines into account during consultations. The ASTI critiquing module is one such systems; it provides the physician with automatic criticisms when a drug prescription does not follow the guidelines. It was initially developed for hypertension and type 2 diabetes, but is designed to be generic enough for application to all chronic diseases. We present here the results of usability and satisfaction evaluations for the ASTI critiquing module, obtained with GPs for a newly implemented guideline concerning…
Evaluating Compliance with Visualization Guidelines in Diagrams for Scientific Publications Using Large Vision Language Models
Department of Computer Science, University of Applied Sciences and Arts Dortmund（应用科学与艺术大学多特蒙德计算机科学系）；Institute for Medical Informatics, Biometry and Epidemiology (IMIBE)（医学信息学、生物统计与流行病学研究所）；Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen（医学人工智能研究所，埃森大学医院）
Institute for Computing and Information Sciences（计算与信息科学研究所）；Radboud University Nijmegen（拉德堡德大学奈梅亨分校）
Requirements about the quality of clinical guidelines can be represented by schemata borrowed from the theory of abductive diagnosis, using temporal logic to model the time-oriented aspects expressed in a guideline. Previously, we have shown that these requirements can be verified using interactive theorem proving techniques. In this paper, we investigate how this approach can be mapped to the facilities of a resolution-based theorem prover, Otter, and a complementary program that searches for finite models of first-order statements, Mace. It is shown that the reasoning required for checking t…
LLM-Orchestrated Conformance Checking in Stroke Care Without Computer-Interpretable Guidelines
Computer Science Institute, DiSIT, University of Piemonte Orientale（皮埃蒙特东方大学计算机科学研究所）；Integrated Laboratory of AI and Medical Informatics, DAIRI, SS. Antonio e Biagio e Cesare Arrigo Hospital（圣安东尼奥、比亚焦与切萨雷·阿里戈医院DAIRI人工智能与医学信息学综合实验室）
National Technical University of Athens（雅典国立技术大学）；University of Athens（雅典大学）
Cardiovascular Disease (CVD) is an important cause of disability and death among individuals with Diabetes Mellitus (DM). International clinical guidelines for the management of Type 2 DM (T2DM) are founded on primary and secondary prevention and favor the evaluation of CVD related risk factors towards appropriate treatment initiation. CVD risk prediction models can provide valuable tools for optimizing the frequency of medical visits and performing timely preventive and therapeutic interventions against CVD events. The integration of explainability modalities in these models can enhance human…
Propagating construction-time knowledge quality into medical question answering: A framework grounded in clinical guidelines
Nanjing Medical University（南京医科大学）；The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院）；The Friendship Hospital of Ili Kazakh Autonomous Prefecture（伊犁哈萨克自治州友谊医院）；Nanjing University（南京大学）
Medical experts may use Artificial Intelligence (AI) systems with greater trust if these are supported by contextual explanations that let the practitioner connect system inferences to their context of use. However, their importance in improving model usage and understanding has not been extensively studied. Hence, we consider a comorbidity risk prediction scenario and focus on contexts regarding the patients clinical state, AI predictions about their risk of complications, and algorithmic explanations supporting the predictions. We explore how relevant information for such dimensions can be e…
Management of chronic diseases such as heart failure, diabetes, and chronic obstructive pulmonary disease (COPD) is a major problem in health care. A standard approach that the medical community has devised to manage widely prevalent chronic diseases such as chronic heart failure (CHF) is to have a committee of experts develop guidelines that all physicians should follow. These guidelines typically consist of a series of complex rules that make recommendations based on a patient's information. Due to their complexity, often the guidelines are either ignored or not complied with at all, which c…
Identification of Regulatory Requirements Relevant to Business Processes: A Comparative Study on Generative AI, Embedding-based Ranking, Crowd and Expert-driven Methods
Technical University of Munich（慕尼黑工业大学）；University of Queensland（昆士兰大学）
