The Coronavirus (COVID-19) pandemic has been the research focus world-wide in the year 2020. Several efforts, from collection of COVID-19 patients' data to screening them for the virus's detection are taken with rigour. A major portion of COVID-19 symptoms are related to the functioning of the respiratory system, which in-turn critically influences the human speech production system. This drives the research focus towards identifying the markers of COVID-19 in speech and other human generated audio signals. In this paper, we give an overview of the speech and other audio signal, language and general signal processing-based work done using Artificial Intelligence techniques to screen, diagnose, monitor, and spread the awareness aboutCOVID-19. We also briefly describe the research related to detect accord-ing COVID-19 symptoms carried out so far. We aspire that this collective information will be useful in developing automated systems, which can help in the context of COVID-19 using non-obtrusive and easy to use modalities such as audio, speech, and language.
Imperial College London(伦敦帝国理工学院);North Bristol NHS Trust(北布里斯托尔国民健康服务体系信托基金);University of Augsburg(奥格斯堡大学)
We report on cross-running the recent COVID-19 Identification ResNet (CIdeR) on the two Interspeech 2021 COVID-19 diagnosis from cough and speech audio challenges: ComParE and DiCOVA. CIdeR is an end-to-end deep learning neural network originally designed to classify whether an individual is COVID-positive or COVID-negative based on coughing and breathing audio recordings from a published crowdsourced dataset. In the current study, we demonstrate the potential of CIdeR at binary COVID-19 diagnosis from both the COVID-19 Cough and Speech Sub-Challenges of INTERSPEECH 2021, ComParE and DiCOVA. C…
Imperial College London(伦敦帝国理工学院);University of Augsburg(奥格斯堡大学);North Bristol NHS Trust(北布里斯托尔国民保健服务信托基金)
Our main contributions are as follows: (I) We demonstrate the first attempt to diagnose COVID-19 using end-to-end deep learning from a crowd-sourced dataset of audio samples, achieving ROC-AUC of 0.846; (II) Our model, the COVID-19 Identification ResNet, (CIdeR), has potential for rapid scalability, minimal cost and improving performance as more data becomes available. This could enable regular COVID-19 testing at apopulation scale; (III) We introduce a novel modelling strategy using a custom deep neural network to diagnose COVID-19 from a joint breath and cough representation; (IV) We release…
University of Brighton(布莱顿大学);Royal Holloway University of London(伦敦大学皇家霍洛威学院)
Audio classification using breath and cough samples has recently emerged as a low-cost, non-invasive, and accessible COVID-19 screening method. However, a comprehensive survey shows that no application has been approved for official use at the time of writing, due to the stringent reliability and accuracy requirements of the critical healthcare setting. To support the development of Machine Learning classification models, we performed an extensive comparative investigation and ranking of 15 audio features, including less well-known ones. The results were verified on two independent COVID-19 so…
University of Cambridge(剑桥大学);University of Southampton(南安普顿大学)
Recent work has shown the potential of using audio data (eg, cough, breathing, and voice) in the screening for COVID-19. However, these approaches only focus on one-off detection and detect the infection given the current audio sample, but do not monitor disease progression in COVID-19. Limited exploration has been put forward to continuously monitor COVID-19 progression, especially recovery, through longitudinal audio data. Tracking disease progression characteristics could lead to more timely treatment. The primary objective of this study is to explore the potential of longitudinal audio sam…
University of Cambridge(剑桥大学);University of Southampton(南安普顿大学)
Researchers have been battling with the question of how we can identify Coronavirus disease (COVID-19) cases efficiently, affordably and at scale. Recent work has shown how audio based approaches, which collect respiratory audio data (cough, breathing and voice) can be used for testing, however there is a lack of exploration of how biases and methodological decisions impact these tools' performance in practice. In this paper, we explore the realistic performance of audio-based digital testing of COVID-19. To investigate this, we collected a large crowdsourced respiratory audio dataset through …
Fine-tuning Pre-trained Audio Models for COVID-19 Detection: A Technical Report
Institute of Biosciences, Languages, and Exact Sciences(生物科学、语言和精确科学学院);São Paulo State University(圣保罗州大学);Institute of Natural Sciences and Engineering(自然科学与工程学院);University of São Paulo(圣保罗大学);Institute of Mathematics and Statistics(数学与统计学院)
This work explores speech as a biomarker and investigates the detection of respiratory insufficiency (RI) by analyzing speech samples. Previous work \cite{spira2021} constructed a dataset of respiratory insufficiency COVID-19 patient utterances and analyzed it by means of a convolutional neural network achieving an accuracy of $87.04\%$, validating the hypothesis that one can detect RI through speech. Here, we study how Transformer neural network architectures can improve the performance on RI detection. This approach enables construction of an acoustic model. By choosing the correct pretraini…
Saw Swee Hock School of Public Health, National University of Singapore(新加坡国立大学苏瑞福公共卫生学院);National University Health System, National University of Singapore(新加坡国立大学国立大学医疗系统);National University Hospital, Singapore(新加坡国立大学医院);Yong Loo Lin Medical School, National University of Singapore(新加坡国立大学杨潞龄医学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究所)
We suggested a unified system with core components of data augmentation, ImageNet-pretrained ResNet-50, cost-sensitive loss, deep ensemble learning, and uncertainty estimation to quickly and consistently detect COVID-19 using acoustic evidence. To increase the model's capacity to identify a minority class, data augmentation and cost-sensitive loss are incorporated (infected samples). In the COVID-19 detection challenge, ImageNet-pretrained ResNet-50 has been found to be effective. The unified framework also integrates deep ensemble learning and uncertainty estimation to integrate predictions f…
Virufy AI Research Group(Virufy AI研究组);University of Alexandria(亚历山大大学);Duke University(杜克大学);University of California San Francisco(加利福尼亚大学旧金山分校);Stanford University(斯坦福大学);University of California Berkeley(加利福尼亚大学伯克利分校)
Fast and affordable solutions for COVID-19 testing are necessary to contain the spread of the global pandemic and help relieve the burden on medical facilities. Currently, limited testing locations and expensive equipment pose difficulties for individuals trying to be tested, especially in low-resource settings. Researchers have successfully presented models for detecting COVID-19 infection status using audio samples recorded in clinical settings [5, 15], suggesting that audio-based Artificial Intelligence models can be used to identify COVID-19. Such models have the potential to be deployed o…
Institute for Informatics and Telematics of the National Research Council of Italy(意大利国家研究委员会信息学与远程通信研究所);University of Milano(米兰大学)
Smartphones and wearable devices, along with Artificial Intelligence, can represent a game-changer in the pandemic control, by implementing low-cost and pervasive solutions to recognize the development of new diseases at their early stages and by potentially avoiding the rise of new outbreaks. Some recent works show promise in detecting diagnostic signals of COVID-19 from voice and coughs by using machine learning and hand-crafted acoustic features. In this paper, we decided to investigate the capabilities of the recently proposed deep embedding model L3-Net to automatically extract meaningful…
The University of Tokyo(东京大学);Imperial College London(帝国理工学院)
Computer audition (CA) has been demonstrated to be efficient in healthcare domains for speech-affecting disorders (e.g., autism spectrum, depression, or Parkinson's disease) and body sound-affecting abnormalities (e. g., abnormal bowel sounds, heart murmurs, or snore sounds). Nevertheless, CA has been underestimated in the considered data-driven technologies for fighting the COVID-19 pandemic caused by the SARS-CoV-2 coronavirus. In this light, summarise the most recent advances in CA for COVID-19 speech and/or sound analysis. While the milestones achieved are encouraging, there are yet not an…
Secretaría de Innovavión y Transformación Digital(创新与数字化转型秘书处);Subsecretaría de Planificación Sanitaria(卫生规划副秘书处);Ministerio de Salud(卫生部)
OBJECTIVE: Our objective is to evaluate the possibility of using cough audio recordings (spontaneous or simulated) to detect sound patterns in people who are diagnosed with COVID-19. The research question that led our work was: what is the sensitivity and specificity of a machine learning based COVID-19 cough classifier, using RT-PCR tests as gold standard? SETTING: The audio samples that were collected for this study belong to individuals who were swabbed in the City of Buenos Aires in 20 public and 1 private facilities where RT-PCR studies were carried out on patients suspected of COVID, and…
University of Essex(埃塞克斯大学);Austrian Institute of Technology(奥地利技术研究所);FPT Software Company Limited(FPT软件有限公司)
This paper evaluates a wide range of audio-based deep learning frameworks applied to the breathing, cough, and speech sounds for detecting COVID-19. In general, the audio recording inputs are transformed into low-level spectrogram features, then they are fed into pre-trained deep learning models to extract high-level embedding features. Next, the dimension of these high-level embedding features are reduced before finetuning using Light Gradient Boosting Machine (LightGBM) as a back-end classification. Our experiments on the Second DiCOVA Challenge achieved the highest Area Under the Curve (AUC…
The Alan Turing Institute(阿兰·图灵研究所);Imperial College London(帝国理工学院);University of Oxford(牛津大学);University of Surrey(萨里大学);King’s College London(伦敦国王学院);University College London(伦敦大学学院);UK Health Security Agency(英国卫生安全局);University of Brighton(布莱顿大学)
Recent work has reported that AI classifiers trained on audio recordings can accurately predict severe acute respiratory syndrome coronavirus 2 (SARSCoV2) infection status. Here, we undertake a large scale study of audio-based deep learning classifiers, as part of the UK governments pandemic response. We collect and analyse a dataset of audio recordings from 67,842 individuals with linked metadata, including reverse transcription polymerase chain reaction (PCR) test outcomes, of whom 23,514 tested positive for SARS CoV 2. Subjects were recruited via the UK governments National Health Service T…
TCS Research India(塔塔咨询服务公司印度研究院);University of Augsburg(奥格斯堡大学);Imperial College London(帝国理工学院)
Recently, there has been an increased attention towards innovating, enhancing, building, and deploying applications of speech signal processing for providing assistance and relief to human mankind from the Coronavirus (COVID-19) pandemic. Many AI with speech initiatives are taken to combat with the present situation and also to create a safe and secure environment for the future. This paper summarises all these efforts taken by the re-search community towards helping the individuals and the society in the fight against COVID-19 over the past 3-4 months using speech signal processing. We also s…
We apply the vision transformer, a deep machine learning model build around the attention mechanism, on mel-spectrogram representations of raw audio recordings. When adding mel-based data augmentation techniques and sample-weighting, we achieve comparable performance on both (PRS and CCS challenge) tasks of ComParE21, outperforming most single model baselines. We further introduce overlapping vertical patching and evaluate the influence of parameter configurations. Index Terms: audio classification, attention, mel-spectrogram, unbalanced data-sets, computational paralinguistics
Virufy AI Research Group(Virufy AI研究组);University of Alexandria(亚历山大大学);Stanford University(斯坦福大学);Duke University(杜克大学)
Rapid and affordable methods of testing for COVID-19 infections are essential to reduce infection rates and prevent medical facilities from becoming overwhelmed. Current approaches of detecting COVID-19 require in-person testing with expensive kits that are not always easily accessible. This study demonstrates that crowdsourced cough audio samples recorded and acquired on smartphones from around the world can be used to develop an AI-based method that accurately predicts COVID-19 infection with an ROC-AUC of 77.1% (75.2%-78.3%). Furthermore, we show that our method is able to generalize to cro…
A large-scale and PCR-referenced vocal audio dataset for COVID-19
London Centre for Nanotechnology, University College London(伦敦纳米技术中心,伦敦大学学院);Division of Medicine, University College London(医学系,伦敦大学学院);King's College London, UK(伦敦国王学院);The Alan Turing Institute, UK(阿兰·图灵研究所);Imperial College London, UK(帝国理工学院);UK Health Security Agency, UK(英国卫生安全局);Institute of Health Informatics, University College London, UK(健康信息学研究所,伦敦大学学院);Centre for Stress and Age-Related Disease, School of Applied Sciences, University of Brighton, UK(布莱顿大学应用科学学院压力与年龄相关疾病中心);University of Oxford, UK(牛津大学);University of Surrey, UK(萨里大学);University of Lancaster, UK(兰卡斯特大学);University of Cambridge, UK(剑桥大学);University of Augsburg, Germany(奥格斯堡大学);University of Nottingham, UK(诺丁汉大学)
In this paper, we describe an approach for representation learning of audio signals for the task of COVID-19 detection. The raw audio samples are processed with a bank of 1-D convolutional filters that are parameterized as cosine modulated Gaussian functions. The choice of these kernels allows the interpretation of the filterbanks as smooth band-pass filters. The filtered outputs are pooled, log-compressed and used in a self-attention based relevance weighting mechanism. The relevance weighting emphasizes the key regions of the time-frequency decomposition that are important for the downstream…
University of Augsburg(奥格斯堡大学);Imperial College London(帝国理工学院)
The COVID-19 pandemic has affected the world unevenly; while industrial economies have been able to produce the tests necessary to track the spread of the virus and mostly avoided complete lockdowns, developing countries have faced issues with testing capacity. In this paper, we explore the usage of deep learning models as a ubiquitous, low-cost, pre-testing method for detecting COVID-19 from audio recordings of breathing or coughing taken with mobile devices or via the web. We adapt an ensemble of Convolutional Neural Networks that utilise raw breathing and coughing audio and spectrograms to …
Imperial College London(伦敦帝国学院);University of Augsburg(奥格斯堡大学);audEERING GmbH(audEERING有限公司);The University of Tokyo(东京大学)
At the time of writing, the world population is suffering from more than 10,000 registered COVID-19 disease epidemic induced deaths since the outbreak of the Corona virus more than three months ago now officially known as SARS-CoV-2. Since, tremendous efforts have been made worldwide to counter-steer and control the epidemic by now labelled as pandemic. In this contribution, we provide an overview on the potential for computer audition (CA), i.e., the usage of speech and sound analysis by artificial intelligence to help in this scenario. We first survey which types of related or contextually s…
University of Augsburg(奥格斯堡大学);Johannes Kepler University Linz(约翰内斯·开普勒林茨大学);Imperial College London(帝国理工学院)
Detecting COVID-19 from audio signals, such as breathing and coughing, can be used as a fast and efficient pre-testing method to reduce the virus transmission. Due to the promising results of deep learning networks in modelling time sequences, and since applications to rapidly identify COVID in-the-wild should require low computational effort, we present a temporal-oriented broadcasting residual learning method that achieves efficient computation and high accuracy with a small model size. Based on the EfficientNet architecture, our novel network, named Temporal-oriented ResNet~(TorNet), consti…
Stellenbosch University(斯泰伦博斯大学);SAMRC Centre for Tuberculosis Research(南非医学研究理事会结核病研究中心)
We present an experimental investigation into the effectiveness of transfer learning and bottleneck feature extraction in detecting COVID-19 from audio recordings of cough, breath and speech. This type of screening is non-contact, does not require specialist medical expertise or laboratory facilities and can be deployed on inexpensive consumer hardware. We use datasets that contain recordings of coughing, sneezing, speech and other noises, but do not contain COVID-19 labels, to pre-train three deep neural networks: a CNN, an LSTM and a Resnet50. These pre-trained networks are subsequently eith…
University of Cambridge(剑桥大学);Alan Turing Institute(艾伦·图灵研究所)
Audio signals generated by the human body (e.g., sighs, breathing, heart, digestion, vibration sounds) have routinely been used by clinicians as indicators to diagnose disease or assess disease progression. Until recently, such signals were usually collected through manual auscultation at scheduled visits. Research has now started to use digital technology to gather bodily sounds (e.g., from digital stethoscopes) for cardiovascular or respiratory examination, which could then be used for automatic analysis. Some initial work shows promise in detecting diagnostic signals of COVID-19 from voice …
King’s College London(伦敦国王学院);The Alan Turing Institute(阿兰·图灵研究所);University College London(伦敦大学学院);UK Health Security Agency(英国卫生安全局);Imperial College London(帝国理工学院);University of Oxford(牛津大学);University of Surrey(萨里大学)
Since early in the coronavirus disease 2019 (COVID-19) pandemic, there has been interest in using artificial intelligence methods to predict COVID-19 infection status based on vocal audio signals, for example cough recordings. However, existing studies have limitations in terms of data collection and of the assessment of the performances of the proposed predictive models. This paper rigorously assesses state-of-the-art machine learning techniques used to predict COVID-19 infection status based on vocal audio signals, using a dataset collected by the UK Health Security Agency. This dataset incl…
Discriminant audio properties in deep learning based respiratory insufficiency detection in Brazilian Portuguese
Universidade de São Paulo(圣保罗大学);Universidade Estadual Paulista(圣保罗州立大学);Center for Artificial Intelligence (C4AI-USP)(人工智能中心(C4AI-USP));IBM Corporation(国际商业机器公司(IBM))
Automated audio captioning is a cross-modal translation task that aims to generate natural language descriptions for given audio clips. This task has received increasing attention with the release of freely available datasets in recent years. The problem has been addressed predominantly with deep learning techniques. Numerous approaches have been proposed, such as investigating different neural network architectures, exploiting auxiliary information such as keywords or sentence information to guide caption generation, and employing different training strategies, which have greatly facilitated …
The Second Diagnosis of COVID-19 using Acoustics (DiCOVA) Challenge aimed at accelerating the research in acoustics based detection of COVID-19, a topic at the intersection of acoustics, signal processing, machine learning, and healthcare. This paper presents the details of the challenge, which was an open call for researchers to analyze a dataset of audio recordings consisting of breathing, cough and speech signals. This data was collected from individuals with and without COVID-19 infection, and the task in the challenge was a two-class classification. The development set audio recordings we…
COVID-19 (coronavirus disease 2019) pandemic caused by SARS-CoV-2 has led to a treacherous and devastating catastrophe for humanity. At the time of writing, no specific antivirus drugs or vaccines are recommended to control infection transmission and spread. The current diagnosis of COVID-19 is done by Reverse-Transcription Polymer Chain Reaction (RT-PCR) testing. However, this method is expensive, time-consuming, and not easily available in straitened regions. An interpretable and COVID-19 diagnosis AI framework is devised and developed based on the cough sounds features and symptoms metadata…
University of Augsburg(奥格斯堡大学);Imperial College London(伦敦帝国理工学院)
Aiming to automatically detect COVID-19 from cough sounds, we propose a deep attentive multi-model fusion system evaluated on the Track-1 dataset of the DiCOVA 2021 challenge. Three kinds of representations are extracted, including hand-crafted features, image-from-audio-based deep representations, and audio-based deep representations. Afterwards, the best models on the three types of features are fused at both the feature level and the decision level. The experimental results demonstrate that the proposed attention-based fusion at the feature level achieves the best performance (AUC: 77.96%) …
An overview of neural architectures for self-supervised audio representation learning from masked spectrograms
Department of Electronic Systems, Aalborg University(电子系统系,奥尔堡大学);Pioneer Centre for Artificial Intelligence(人工智能先锋中心);National and Kapodistrian University of Athens(雅典国家与卡波蒂斯坦大学)
Univ Rennes(雷恩大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);IRISA(法国信息与随机系统研究所);Orange(奥兰治);Univ Lyon(里昂大学);ENS de Lyon(里昂高等师范学院);UCB Lyon 1(里昂第一大学)
Recent advances in audio declipping have substantially improved the state of the art.% in certain saturation regimes. Yet, practitioners need guidelines to choose a method, and while existing benchmarks have been instrumental in advancing the field, larger-scale experiments are needed to guide such choices. First, we show that the clipping levels in existing small-scale benchmarks are moderate and call for benchmarks with more perceptually significant clipping levels. We then propose a general algorithmic framework for declipping that covers existing and new combinations of variants of state-o…
In the break of COVID-19 pandemic, mass testing has become essential to reduce the spread of the virus. Several recent studies suggest that a significant number of COVID-19 patients display no physical symptoms whatsoever. Therefore, it is unlikely that these patients will undergo COVID-19 test, which increases their chances of unintentionally spreading the virus. Currently, the primary diagnostic tool to detect COVID-19 is RT-PCR test on collected respiratory specimens from the suspected case. This requires patients to travel to a laboratory facility to be tested, thereby potentially infectin…
Imperial College London(伦敦帝国理工学院);University of Augsburg(奥格斯堡大学);audEERING GmbH(audEERING公司);University Carlos III of Madrid(马德里卡洛斯三世大学);Medical University of Graz(格拉茨医科大学)
Among the seventeen Sustainable Development Goals (SDGs) proposed within the 2030 Agenda and adopted by all the United Nations member states, the 13$^{th}$ SDG is a call for action to combat climate change for a better world. In this work, we provide an overview of areas in which audio intelligence -- a powerful but in this context so far hardly considered technology -- can contribute to overcome climate-related challenges. We categorise potential computer audition applications according to the five elements of earth, water, air, fire, and aether, proposed by the ancient Greeks in their five e…
University of Augsburg(奥格斯堡大学);Medical University of Graz(格拉茨医科大学);Imperial College(帝国理工学院)
More than two years after its outbreak, the COVID-19 pandemic continues to plague medical systems around the world, putting a strain on scarce resources, and claiming human lives. From the very beginning, various AI-based COVID-19 detection and monitoring tools have been pursued in an attempt to stem the tide of infections through timely diagnosis. In particular, computer audition has been suggested as a non-invasive, cost-efficient, and eco-friendly alternative for detecting COVID-19 infections through vocal sounds. However, like all AI methods, also computer audition is heavily dependent on …
The development of fast and accurate screening tools, which could facilitate testing and prevent more costly clinical tests, is key to the current pandemic of COVID-19. In this context, some initial work shows promise in detecting diagnostic signals of COVID-19 from audio sounds. In this paper, we propose a voice-based framework to automatically detect individuals who have tested positive for COVID-19. We evaluate the performance of the proposed framework on a subset of data crowdsourced from our app, containing 828 samples from 343 participants. By combining voice signals and reported symptom…
Indian Institute of Science(印度科学研究院);P. D. Hinduja Hospital(P. D. 欣杜贾医院);KMC Hospital(KMC 医院)
The DiCOVA challenge aims at accelerating research in diagnosing COVID-19 using acoustics (DiCOVA), a topic at the intersection of speech and audio processing, respiratory health diagnosis, and machine learning. This challenge is an open call for researchers to analyze a dataset of sound recordings collected from COVID-19 infected and non-COVID-19 individuals for a two-class classification. These recordings were collected via crowdsourcing from multiple countries, through a website application. The challenge features two tracks, one focusing on cough sounds, and the other on using a collection…
Cough audio signal classification has been successfully used to diagnose a variety of respiratory conditions, and there has been significant interest in leveraging Machine Learning (ML) to provide widespread COVID-19 screening. However, there is currently no validated database of cough sounds with which to train such ML models. The COUGHVID dataset provides over 20,000 crowdsourced cough recordings representing a wide range of subject ages, genders, geographic locations, and COVID-19 statuses. First, we filtered the dataset using our open-sourced cough detection algorithm. Second, experienced …
The current outbreak of a coronavirus, has quickly escalated to become a serious global problem that has now been declared a Public Health Emergency of International Concern by the World Health Organization. Infectious diseases know no borders, so when it comes to controlling outbreaks, timing is absolutely essential. It is so important to detect threats as early as possible, before they spread. After a first successful DiCOVA challenge, the organisers released second DiCOVA challenge with the aim of diagnosing COVID-19 through the use of breath, cough and speech audio samples. This work prese…
Spoken keyword spotting (KWS) deals with the identification of keywords in audio streams and has become a fast-growing technology thanks to the paradigm shift introduced by deep learning a few years ago. This has allowed the rapid embedding of deep KWS in a myriad of small electronic devices with different purposes like the activation of voice assistants. Prospects suggest a sustained growth in terms of social use of this technology. Thus, it is not surprising that deep KWS has become a hot research topic among speech scientists, who constantly look for KWS performance improvement and computat…
Environmental Sound Deepfake Detection Challenge: An Overview
1 School of Electrical Engineering, KAIST, Daejeon, Republic of Korea 2 University of Melbourne, Australia, 3 Fortemedia Singapore, Singapore, 4 Xi’an University of Posts \& Telecommunications, Xi’an, China, 5 Xi'an Lianfeng Acoustic Technologies Co., Ltd., China
The technology development for point-of-care tests (POCTs) targeting respiratory diseases has witnessed a growing demand in the recent past. Investigating the presence of acoustic biomarkers in modalities such as cough, breathing and speech sounds, and using them for building POCTs can offer fast, contactless and inexpensive testing. In view of this, over the past year, we launched the ``Coswara'' project to collect cough, breathing and speech sound recordings via worldwide crowdsourcing. With this data, a call for development of diagnostic tools was announced in the Interspeech 2021 as a spec…