Publicly accessible benchmarks that allow for assessing and comparing model performances are important drivers of progress in artificial intelligence (AI). While recent advances in AI capabilities hold the potential to transform medical practice by assisting and augmenting the cognitive processes of healthcare professionals, the coverage of clinically relevant tasks by AI benchmarks is largely unclear. Furthermore, there is a lack of systematized meta-information that allows clinical AI researchers to quickly determine accessibility, scope, content and other characteristics of datasets and benchmark datasets relevant to the clinical domain.
To address these issues, we curated and released a comprehensive catalogue of datasets and benchmarks pertaining to the broad domain of clinical and biomedical natural language processing (NLP), based on a systematic review of literature and online resources. A total of 450 NLP datasets were manually systematized and annotated with rich metadata, such as targeted tasks, clinical applicability, data types, performance metrics, accessibility and licensing information, and availability of data splits. We then compared tasks covered by AI benchmark datasets with relevant tasks that medical practitioners reported as highly desirable targets for automation in a previous empirical study.
Our analysis indicates that AI benchmarks of direct clinical relevance are scarce and fail to cover most work activities that clinicians want to see addressed. In particular, tasks associated with routine documentation and patient data administration workflows are not represented despite significant associated workloads. Thus, currently available AI benchmarks are improperly aligned with desired targets for AI automation in clinical settings, and novel benchmarks should be created to fill these gaps.
University of Nottingham（诺丁汉大学）；The University of Nottingham Ningbo（宁波诺丁汉大学）；Nottingham University Hospitals（诺丁汉大学医院）
In the context of cancer treatment and surgery, quality of life assessment is a crucial part of determining treatment success and viability. In order to assess it, patients completed questionnaires which employ words to capture aspects of patients well-being are the norm. As the results of these questionnaires are often used to assess patient progress and to determine future treatment options, it is important to establish that the words used are interpreted in the same way by both patients and medical professionals. In this paper, we capture and model patients perceptions and associated uncert…
National Institute for Mathematical Sciences（韩国数学科学研究所）
Deep learning technology is making great progress in solving the challenging problems of artificial intelligence, hence machine learning based on artificial neural networks is in the spotlight again. In some areas, artificial intelligence based on deep learning is beyond human capabilities. It seemed extremely difficult for a machine to beat a human in a Go game, but AlphaGo has shown to beat a professional player in the game. By looking at the statistical distribution of the distance in which the Go stones are laid in succession, we find a clear trace that Alphago has surpassed human abilitie…
Kyiv National University of Construction and Architecture（基辅国立建筑大学）
This study is devoted to solving the problem to determine the professional adaptive capabilities of construction management staff using artificial intelligence systems.It is proposed Fully Connected Feed-Forward Neural Network architecture and performed empirical modeling to create a Data Set. Model of artificial intelligence system allows evaluating the processes in an Fully Connected Feed-Forward Neural Network during the execution of multi-value classification of professional areas. A method has been developed for the training process of a machine learning model, which reflects the internal…
Artificial General Intelligence for Medical Imaging Analysis
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；The University of Georgia（佐治亚大学）；Beijing Jiaotong University（北京交通大学）；Chinese University of Hong Kong（香港中文大学）；Second Xiangya Hospital（湘雅二医院）
CBIDR: A novel method for information retrieval combining image and data by means of TOPSIS applied to medical diagnosis
Labcin - Nature-inspired computing Lab. - Federal University of Espírito Santo - Vitória, ES, Brazil（自然启发计算实验室 - 芬斯大学 - 维蒂利亚）；Graduate Program in Computer Science, PPGI, UFES - Federal University of Espírito Santo - Vitória, ES, Brazil（计算机科学研究生项目，PPGI，UFES - 芬斯大学 - 维蒂利亚）
Martin Centre for Architectural and Urban Studies, University of Cambridge（剑桥大学马丁建筑与城市研究中心）；Department of Architecture, University of Cambridge（剑桥大学建筑系）
Deep learning applications in shaping ad hoc planning proposals are limited by the difficulty in integrating professional knowledge about cities with artificial intelligence. We propose a novel, complementary use of deep neural networks and planning guidance to automate street network generation that can be context-aware, example-based and user-guided. The model tests suggest that the incorporation of planning knowledge (e.g., road junctions and neighborhood types) in the model training leads to a more realistic prediction of street configurations. Furthermore, the new tool provides both profe…
Colorado State University（科罗拉多州立大学）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）
Computing systems are tightly integrated today into our professional, social, and private lives. An important consequence of this growing ubiquity of computing is that it can have significant ethical implications of which computing professionals should take account. In most real-world scenarios, it is not immediately obvious how particular technical choices during the design and use of computing systems could be viewed from an ethical perspective. This article provides a perspective on the ethical challenges within semiconductor chip design, IoT applications, and the increasing use of artifici…
Nanyang Technological University（南洋理工大学）；Alibaba Group Holding Limited（阿里巴巴集团控股有限公司）
Recently, artificial intelligence and machine learning in general have demonstrated remarkable performances in many tasks, from image processing to natural language processing, especially with the advent of deep learning. Along with research progress, they have encroached upon many different fields and disciplines. Some of them require high level of accountability and thus transparency, for example the medical sector. Explanations for machine decisions and predictions are thus needed to justify their reliability. This requires greater interpretability, which often means we need to understand t…
University of Maribor（马里博尔大学）；Dr. Adolf Drolc Healthcare Centre（阿道夫·德罗尔茨医生医疗中心）
The use of artificial intelligence intelligencein medicine can be traced back to 1968 when Paycha published his paper Le diagnostic a l'aide d'intelligences artificielle, presentation de la premiere machine diagnostri. Few years later Shortliffe et al. presented an expert system named Mycin which was able to identify bacteria causing severe blood infections and to recommend antibiotics. Despite the fact that Mycin outperformed members of the Stanford medical school in the reliability of diagnosis it was never used in practice due to a legal issue who do you sue if it gives a wrong diagnosis?. …
Mayo Clinic（梅奥诊所）；the First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）；University of Georgia（佐治亚大学）；University of Electronic Science and Technology of China（电子科技大学）；Eye Institute of Xiamen University（厦门大学眼科研究所）
Pontifical Catholic University of Rio Grande do Sul（南里奥格兰德天主教大学）
The 4th Industrial Revolution is the culmination of the digital age. Nowadays, technologies such as robotics, nanotechnology, genetics, and artificial intelligence promise to transform our world and the way we live. Artificial Intelligence Ethics and Safety is an emerging research field that has been gaining popularity in recent years. Several private, public and non-governmental organizations have published guidelines proposing ethical principles for regulating the use and development of autonomous intelligent systems. Meta-analyses of the AI Ethics research field point to convergence on cert…
Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics
ZBeats Inc（ZBeats公司）；New York University（纽约大学）；Stony Brook Medicine（石溪医学中心）；University of Pennsylvania（宾夕法尼亚大学）；Binghamton University（宾汉姆顿大学）；Microsoft（微软公司）
University of Maryland, Baltimore County（马里兰大学巴尔的摩郡分校）；University of North Carolina, Wilmington（北卡罗来纳大学威尔明顿分校）
Cyber-defense systems are being developed to automatically ingest Cyber Threat Intelligence (CTI) that contains semi-structured data and/or text to populate knowledge graphs. A potential risk is that fake CTI can be generated and spread through Open-Source Intelligence (OSINT) communities or on the Web to effect a data poisoning attack on these systems. Adversaries can use fake CTI examples as training input to subvert cyber defense systems, forcing the model to learn incorrect inputs to serve their malicious needs. In this paper, we automatically generate fake CTI text descriptions using tran…
Ontology-based knowledge representation for bone disease diagnosis: a foundation for safe and sustainable medical artificial intelligence systems
University of Science, VNUHCM（胡志明市国家大学科学大学）；Viet Nam National University, Ho Chi Minh City（越南国家大学胡志明市分校）；Ho Chi Minh City Hospital for Traumatology and Orthopedics（胡志明市创伤与骨科医院）
Although machine learning (ML) models of AI achieve high performances in medicine, they are not free of errors. Empowering clinicians to identify incorrect model recommendations is crucial for engendering trust in medical AI. Explainable AI (XAI) aims to address this requirement by clarifying AI reasoning to support the end users. Several studies on biomedical imaging achieved promising results recently. Nevertheless, solutions for models using tabular data are not sufficient to meet the requirements of clinicians yet. This paper proposes a methodology to support clinicians in identifying fail…
Domain-aware machine learning (ML) models have been increasingly adopted for accelerating small molecule therapeutic design in the recent years. These models have been enabled by significant advancement in state-of-the-art artificial intelligence (AI) and computing infrastructures. Several ML architectures are pre-dominantly and independently used either for predicting the properties of small molecules, or for generating lead therapeutic candidates. Synergetically using these individual components along with robust representation and data generation techniques autonomously in closed loops hold…
University of California Santa Cruz（加州大学圣克鲁兹分校）；Weill Cornell Medicine（威尔康奈尔医学院）；Penn State University（宾夕法尼亚州立大学）
Recently there has been a huge interest in dialog systems. This interest has also been developed in the field of the medical domain where researchers are focusing on building a dialog system in the medical domain. This research is focused on the multi-turn dialog system trained on the multi-turn dialog data. It is difficult to gather a huge amount of multi-turn conversational data in the medical domain that is verified by professionals and can be trusted. However, there are several frequently asked questions (FAQs) or single-turn QA pairs that have information that is verified by the experts a…
What is Implementation Science; and Why It Matters for Bridging the Artificial Intelligence Innovation-to-Application Gap in Medical Imaging
Tehran University of Medical Sciences（德黑兰医科大学）；Imperial College（帝国理工学院）；Washington University in St. Louis（圣路易斯华盛顿大学）；Sultan Qaboos University（苏丹卡布斯大学）；Universiti Sains Malaysia（马来西亚理科大学）；University of British Columbia（不列颠哥伦比亚大学）；BC Cancer Research Institute（不列颠哥伦比亚癌症研究所）
Aalto University（阿尔托大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
In the present paper we present the potential of Explainable Artificial Intelligence methods for decision-support in medical image analysis scenarios. With three types of explainable methods applied to the same medical image data set our aim was to improve the comprehensibility of the decisions provided by the Convolutional Neural Network (CNN). The visual explanations were provided on in-vivo gastral images obtained from a Video capsule endoscopy (VCE), with the goal of increasing the health professionals' trust in the black box predictions. We implemented two post-hoc interpretable machine l…
Institute of Computing, Universidade Federal do Amazonas（亚马逊联邦大学计算学院）；School of Business, University of South-Eastern Norway（东南挪威大学商学院）
Artificial Intelligence (AI) or Machine Learning (ML) systems have been widely adopted as value propositions by companies in all industries in order to create or extend the services and products they offer. However, developing AI/ML systems has presented several engineering problems that are different from those that arise in, non-AI/ML software development. This study aims to investigate how software engineering (SE) has been applied in the development of AI/ML systems and identify challenges and practices that are applicable and determine whether they meet the needs of professionals. Also, w…
Harvard Medical School（哈佛医学院）；Aetna, CVS Health（安泰保险，CVS健康公司）
Health information is generally fragmented across silos. Though it is technically feasible to unite data for analysis in a manner that underpins a rapid learning healthcare system, privacy concerns and regulatory barriers limit data centralization. Machine learning can be conducted in a federated manner on patient datasets with the same set of variables, but separated across sites of care. But federated learning cannot handle the situation where different data types for a given patient are separated vertically across different organizations and when patient ID matching across different institu…
One Patient, Many Contexts: Scaling Medical AI with Contextual Intelligence
Harvard Medical School（哈佛医学院）；Clalit Research Institute（Clalit研究所）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Data Science Initiative（哈佛数据科学计划）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）；Clalit Health Services（Clalit医疗服务集团）；Ben Gurion University（本-古里安大学）；Ben Gurion University of the Negev（内盖夫本-古里安大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard University（哈佛大学）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学布罗德研究所）
Applying state-of-the-art machine learning and natural language processing on approximately one million of teleconsultation records, we developed a triage system, now certified and in use at the largest European telemedicine provider. The system evaluates care alternatives through interactions with patients via a mobile application. Reasoning on an initial set of provided symptoms, the triage application generates AI-powered, personalized questions to better characterize the problem and recommends the most appropriate point of care and time frame for a consultation. The underlying technology w…
Several policy options exist, or have been proposed, to further responsible artificial intelligence (AI) development and deployment. Institutions, including U.S. government agencies, states, professional societies, and private and public sector businesses, are well positioned to implement these policies. However, given limited resources, not all policies can or should be equally prioritized. We define and review nine suggested policies for furthering responsible AI, rank each policy on potential use and impact, and recommend prioritization relative to each institution type. We find that pre-de…
Université de Toulouse – INSA（图卢兹大学-法国国立应用科学学院）；Institut de Mathématiques de Toulouse（图卢兹数学研究所）；IRT Saint Exupéry（圣埃克苏佩里技术研究院）
Big data, data science, deep learning, artificial intelligence are the key words of intense hype related with a job market in full evolution, that impose to adapt the contents of our university professional trainings. Which artificial intelligence is mostly concerned by the job offers? Which methodologies and technologies should be favored in the training programs? Which objectives, tools and educational resources do we needed to put in place to meet these pressing needs? We answer these questions in describing the contents and operational resources in the Data Science orientation of the speci…
AKASA, Inc.（AKASA公司）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）
The medical codes prediction problem from clinical notes has received substantial interest in the NLP community, and several recent studies have shown the state-of-the-art (SOTA) code prediction results of full-fledged deep learning-based methods. However, most previous SOTA works based on deep learning are still in early stages in terms of providing textual references and explanations of the predicted codes, despite the fact that this level of explainability of the prediction outcomes is critical to gaining trust from professional medical coders. This raises the important question of how well…
Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases
