Emotions are fundamental to the creation and perception of music performances. However, achieving human-like expression and emotion through machine learning models for performance rendering remains a challenging task. In this work, we present SyMuPe, a novel framework for developing and training affective and controllable symbolic piano performance models. Our flagship model, PianoFlow, uses conditional flow matching trained to solve diverse multi-mask performance inpainting tasks. By design, it supports both unconditional generation and infilling of music performance features. For training, we use a curated, cleaned dataset of 2,968 hours of aligned musical scores and expressive MIDI performances. For text and emotion control, we integrate a piano performance emotion classifier and tune PianoFlow with the emotion-weighted Flan-T5 text embeddings provided as conditional inputs. Objective and subjective evaluations against transformer-based baselines and existing models show that PianoFlow not only outperforms other approaches, but also achieves performance quality comparable to that of human-recorded and transcribed MIDI samples. For emotion control, we present and analyze samples generated under different text conditioning scenarios. The developed model can be integrated into interactive applications, contributing to the creation of more accessible and engaging music performance systems.
AffectMachine-Classical: A novel system for generating affective classical music
Yong Siew Toh Conservatory of Music, National University of Singapore（新加坡国立大学杨秀桃音乐学院）；Centre for Music and Health, National University of Singapore（新加坡国立大学音乐与健康中心）；Augmented Human Lab, DISA, National University of Singapore（新加坡国立大学DISA增强人类实验室）
LARA-Gen: Enabling Continuous Emotion Control for Music Generation Models via Latent Affective Representation Alignment
X-LANCE Lab, Shanghai Jiao Tong University, China（上海交通大学X-LANCE实验室）；Shanghai AI Lab, China（上海人工智能实验室）；Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, China（上海交通大学医学院精神卫生中心）
Within a Music Information Retrieval perspective, the goal of the study presented here is to investigate the impact on sound features of the musician's affective intention, namely when trying to intentionally convey emotional contents via expressiveness. A preliminary experiment has been performed involving $10$ tuba players. The recordings have been analysed by extracting a variety of features, which have been subsequently evaluated by combining both classic and machine learning statistical techniques. Results are reported and discussed.
The field of Automatic Music Generation has seen significant progress thanks to the advent of Deep Learning. However, most of these results have been produced by unconditional models, which lack the ability to interact with their users, not allowing them to guide the generative process in meaningful and practical ways. Moreover, synthesizing music that remains coherent across longer timescales while still capturing the local aspects that make it sound ``realistic'' or ``human-like'' is still challenging. This is due to the large computational requirements needed to work with long sequences of …
Adobe Research（奥多比研究院）；University of Warwick（华威大学）
We introduce the problem of learning affective correspondence between audio (music) and visual data (images). For this task, a music clip and an image are considered similar (having true correspondence) if they have similar emotion content. In order to estimate this crossmodal, emotion-centric similarity, we propose a deep neural network architecture that learns to project the data from the two modalities to a common representation space, and performs a binary classification task of predicting the affective correspondence (true or false). To facilitate the current study, we construct a large s…
Automatic music generation systems have gained in popularity and sophistication as advances in cloud computing have enabled large-scale complex computations such as deep models and optimization algorithms on personal devices. Yet, they still face an important challenge, that of long-term structure, which is key to conveying a sense of musical coherence. We present the MorpheuS music generation system designed to tackle this problem. MorpheuS' novel framework has the ability to generate polyphonic pieces with a given tension profile and long- and short-term repeated pattern structures. A mathem…
Doctoral Program in Electrical and Computer Engineering（电子与计算机工程博士项目）；Faculty of Engineering, University of Porto（葡萄牙波尔图大学工程学院）；Faculty of Science and Technology, University of Coimbra（科英布拉大学科学与技术学院）；Frost School of Music, University of Miami（迈阿密大学弗罗斯特音乐学院）；Faculty of Science and Technology, NOVA University Lisbon（里斯本NOVA大学科学与技术学院）；ISEP, Polytechnic of Porto（波尔图理工学院ISEP）
Columbia University（哥伦比亚大学）；Data Science Institute, Columbia University（哥伦比亚大学数据科学研究所）
Affective responses to music are highly personal. Despite consensus that idiosyncratic factors play a key role in regulating how listeners emotionally respond to music, precisely measuring the marginal effects of these variables has proved challenging. To address this gap, we develop computational methods to measure affective responses to music from over 403M listener comments on a Chinese social music platform. Building on studies from music psychology in systematic and quasi-causal analyses, we test for musical, lyrical, contextual, demographic, and mental health effects that drive listener …
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Goethe University Frankfurt（法兰克福大学）
Although media content is increasingly produced, distributed, and consumed in multiple combinations of modalities, how individual modalities contribute to the perceived emotion of a media item remains poorly understood. In this paper we present MusicVideos (MuVi), a novel dataset for affective multimedia content analysis to study how the auditory and visual modalities contribute to the perceived emotion of media. The data were collected by presenting music videos to participants in three conditions: music, visual, and audiovisual. Participants annotated the music videos for valence and arousal…
University of Southern California（南加利福尼亚大学）；National Technical University of Athens（雅典国家技术大学）
The study of Music Cognition and neural responses to music has been invaluable in understanding human emotions. Brain signals, though, manifest a highly complex structure that makes processing and retrieving meaningful features challenging, particularly of abstract constructs like affect. Moreover, the performance of learning models is undermined by the limited amount of available neuronal data and their severe inter-subject variability. In this paper we extract efficient, personalized affective representations from EEG signals during music listening. To this end, we employ music signals as a …
Affective Music Recommendation: A Rollout-Based World Model for Offline Preference Optimization
LUCID Inc. Toronto Canada；LUCID Inc. Montr\' e al Canada；Mila --- Qu\' e bec AI Institute Montr\' e al Canada；LUCID Inc.；Mila --- Qu\' e bec AI Institute
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Considerable attention has been paid for physiological signal-based emotion recognition in field of affective computing. For the reliability and user friendly acquisition, Electrodermal Activity (EDA) has great advantage in practical applications. However, the EDA-based emotion recognition with hundreds of subjects still lacks effective solution. In this paper, our work makes an attempt to fuse the subject individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the 1-dimensional residual temporal and channel attention network (RTCAN-1D…
