For over 50 years, researchers have been trying to teach computers to read music notation, referred to as Optical Music Recognition (OMR). However, this field is still difficult to access for new researchers, especially those without a significant musical background: few introductory materials are available, and furthermore the field has struggled with defining itself and building a shared terminology. In this tutorial, we address these shortcomings by (1) providing a robust definition of OMR and its relationship to related fields, (2) analyzing how OMR inverts the music encoding process to recover the musical notation and the musical semantics from documents, (3) proposing a taxonomy of OMR, with most notably a novel taxonomy of applications. Additionally, we discuss how deep learning affects modern OMR research, as opposed to the traditional pipeline. Based on this work, the reader should be able to attain a basic understanding of OMR: its objectives, its inherent structure, its relationship to other fields, the state of the art, and the research opportunities it affords.
Zurich University of Applied Sciences（苏黎世应用科学大学）；Ca’Foscari University of Venice（威尼斯大学）
Optical Music Recognition (OMR) is an important and challenging area within music information retrieval, the accurate detection of music symbols in digital images is a core functionality of any OMR pipeline. In this paper, we introduce a novel object detection method, based on synthetic energy maps and the watershed transform, called Deep Watershed Detector (DWD). Our method is specifically tailored to deal with high resolution images that contain a large number of very small objects and is therefore able to process full pages of written music. We present state-of-the-art detection results of …
Optical Music Recognition (OMR) is an important technology within Music Information Retrieval. Deep learning models show promising results on OMR tasks, but symbol-level annotated data sets of sufficient size to train such models are not available and difficult to develop. We present a deep learning architecture called a Convolutional Sequence-to-Sequence model to both move towards an end-to-end trainable OMR pipeline, and apply a learning process that trains on full sentences of sheet music instead of individually labeled symbols. The model is trained and evaluated on a human generated data s…
Previous work has shown that neural architectures are able to perform optical music recognition (OMR) on monophonic and homophonic music with high accuracy. However, piano and orchestral scores frequently exhibit polyphonic passages, which add a second dimension to the task. Monophonic and homophonic music can be described as homorhythmic, or having a single musical rhythm. Polyphonic music, on the other hand, can be seen as having multiple rhythmic sequences, or voices, concurrently. We first introduce a workflow for creating large-scale polyphonic datasets suitable for end-to-end recognition…
University of Alicante（阿利坎特大学）；TU Wien（维也纳工业大学）；Queen Mary University of London（伦敦玛丽女王大学）
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music …
Optical Music Recognition (OMR) has long been without an adequate dataset and ground truth for evaluating OMR systems, which has been a major problem for establishing a state of the art in the field. Furthermore, machine learning methods require training data. We analyze how the OMR processing pipeline can be expressed in terms of gradually more complex ground truth, and based on this analysis, we design the MUSCIMA++ dataset of handwritten music notation that addresses musical symbol recognition and notation reconstruction. The MUSCIMA++ dataset version 0.9 consists of 140 pages of handwritte…
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Considerable attention has been paid for physiological signal-based emotion recognition in field of affective computing. For the reliability and user friendly acquisition, Electrodermal Activity (EDA) has great advantage in practical applications. However, the EDA-based emotion recognition with hundreds of subjects still lacks effective solution. In this paper, our work makes an attempt to fuse the subject individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the 1-dimensional residual temporal and channel attention network (RTCAN-1D…
A Dataset for the Recognition of Historical and Handwritten Music Scores in Western Notation
Computer Vision Center（计算机视觉中心）；Department of Computer Science（计算机科学系）；Department of Arts and Musicology（艺术与音乐学系）；Institute of Formal and Applied Linguistics（形式与应用语言学研究所）；Moravian Library（摩拉维亚图书馆）
Optical Music Recognition (OMR) is concerned with transcribing sheet music into a machine-readable format. The transcribed copy should allow musicians to compose, play and edit music by taking a picture of a music sheet. Complete transcription of sheet music would also enable more efficient archival. OMR facilitates examining sheet music statistically or searching for patterns of notations, thus helping use cases in digital musicology too. Recently, there has been a shift in OMR from using conventional computer vision techniques towards a deep learning approach. In this paper, we review releva…
Full-Page Optical Music Recognition of Handwritten Monophonic Scores
Pattern Recognition and Artificial Intelligence group, University of Alicante（阿利坎特大学模式识别与人工智能组）；Instituto Superior de Enseñanzas Artísticas de la Comunidad Valenciana（瓦伦西亚自治区高等艺术教育机构）
Sheet Music Benchmark: Standardized Optical Music Recognition Evaluation
Pattern Recognition and Artificial Intelligence Group, University of Alicante, Spain（西班牙阿利坎特大学模式识别与人工智能组）；Self-employed（自雇人士）；Center for Computer Research in Music and Acoustics, Stanford University, USA（美国斯坦福大学音乐与声学计算机研究中心）；Instituto Superior de Enseñanzas Artísticas de la Comunidad Valenciana, Spain（西班牙瓦伦西亚自治区高等艺术教育研究所）
提出Sheet Music Benchmark (SMB)数据集和OMR标准化编辑距离(OMR-NED)指标，用于标准化光学乐谱识别(OMR)评估，涵盖多种音乐纹理和细粒度错误分析。
Georgia Institute of Technology（佐治亚理工学院）；Indiana University（印第安纳大学）；EgoVid Inc.（EgoVid公司）
The prevalence of Internet of things (IoT) devices and abundance of sensor data has created an increase in real-time data processing such as recognition of speech, image, and video. While currently such processes are offloaded to the computationally powerful cloud system, a localized and distributed approach is desirable because (i) it preserves the privacy of users and (ii) it omits the dependency on cloud services. However, IoT networks are usually composed of resource-constrained devices, and a single device is not powerful enough to process real-time data. To overcome this challenge, we ex…
Ping An Technology(Shenzhen) Co.,Ltd（平安科技（深圳）有限公司）；University College London（伦敦大学学院）
One of the challenges of the Optical Music Recognition task is to transcript the symbols of the camera-captured images into digital music notations. Previous end-to-end model which was developed as a Convolutional Recurrent Neural Network does not explore sufficient contextual information from full scales and there is still a large room for improvement. We propose an innovative framework that combines a block of Residual Recurrent Convolutional Neural Network with a recurrent Encoder-Decoder network to map a sequence of monophonic music symbols corresponding to the notations present in the ima…
University of Alicante（阿利坎特大学）；TU Wien（维也纳工业大学）；Canemus；Johannes Kepler Universität（约翰·开普勒大学）；Bavarian State Library（巴伐利亚州立图书馆）；McGill University（麦吉尔大学）
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music …
University of Alicante（阿利坎特大学）；TU Wien（维也纳工业大学）；McGill University（麦吉尔大学）；Canamus；Queen Mary University of London（伦敦玛丽女王大学）；Goldsmiths University of London（伦敦大学金史密斯学院）；Computer Vision Center（计算机视觉中心）
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music …
NOTA: Multimodal Music Notation Understanding for Visual Large Language Model
School of Computer Science, Wuhan University（武汉大学计算机学院）；Key Laboratory of Archival Intelligent Development and Service, NAAC（国家档案学院档案智能发展与服务重点实验室）；School of Information Management, Wuhan University（武汉大学信息管理学院）
Seoul National University（首尔大学）；Advanced Institutes of Convergence Technology（融合技术先进研究院）
Identifying musical instruments in polyphonic music recordings is a challenging but important problem in the field of music information retrieval. It enables music search by instrument, helps recognize musical genres, or can make music transcription easier and more accurate. In this paper, we present a convolutional neural network framework for predominant instrument recognition in real-world polyphonic music. We train our network from fixed-length music excerpts with a single-labeled predominant instrument and estimate an arbitrary number of predominant instruments from an audio signal with a…
A High-Accuracy Optical Music Recognition Method Based on Bottleneck Residual Convolutions
Department of Artificial Intelligence, Tianshui Normal University（天水师范学院人工智能系）；Department of Physics, Tianshui Normal University（天水师范学院物理系）；School of Music and Dance, Tianshui Normal University（天水师范学院音乐与舞蹈学院）
Universitat Politècnica de València（瓦伦西亚理工大学）；Charles University（查理大学）；TU Wien（维也纳工业大学）；Conservatoire national des arts et métiers（法国国立工艺学院）；McGill University（麦吉尔大学）；Computer Vision Center（计算机视觉中心）；Johannes Kepler Universität（约翰内斯·开普勒大学）；University of Alicante（阿利坎特大学）；INESC Tec（葡萄牙系统与计算机工程研究所）；Indiana University（印第安纳大学）
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music …
We propose MoodNet - A Deep Convolutional Neural Network based architecture to effectively predict the emotion associated with a piece of music given its audio and lyrical content.We evaluate different architectures consisting of varying number of two-dimensional convolutional and subsampling layers,followed by dense layers.We use Mel-Spectrograms to represent the audio content and word embeddings-specifically 100 dimensional word vectors, to represent the textual content represented by the lyrics.We feed input data from both modalities to our MoodNet architecture.The output from both the moda…
University of Essex（埃塞克斯大学）；Queensland University of Technology（昆士兰科技大学）；University of Southampton（南安普顿大学）
Visual place recognition (VPR) is an essential component of robot navigation and localization systems that allows them to identify a place using only image data. VPR is challenging due to the significant changes in a place's appearance under different illumination throughout the day, with seasonal weather and when observed from different viewpoints. Currently, no single VPR technique excels in every environmental condition, each exhibiting unique benefits and shortcomings. As a result, VPR systems combining multiple techniques achieve more reliable VPR performance in changing environments, at …
