Although perceptual (dis)similarity between sensory stimuli seems akin to distance, measuring the Euclidean distance between vector representations of auditory stimuli is a poor estimator of subjective dissimilarity. In hearing, nonlinear response patterns, interactions between stimulus components, temporal effects, and top-down modulation transform the information contained in incoming frequency-domain stimuli in a way that seems to preserve some notion of distance, but not that of familiar Euclidean space. This work proposes that transformations applied to auditory stimuli during hearing can be modeled as a function mapping stimulus points to their representations in a perceptual space, inducing a Riemannian distance metric. A dataset was collected in a subjective listening experiment, the results of which were used to explore approaches (biologically inspired, data-driven, and combinations thereof) to approximating the perceptual map. Each of the proposed measures achieved comparable or stronger correlations with subjective ratings (r ~ 0.8) compared to state-of-the-art audio quality measures.
We propose a method for an improvement of auditory BCI (aBCI) paradigm based on a combination of ASSR stimuli optimization by choosing the subjects' best responses to AM-, flutter-, AM/FM and click-envelope modulated sounds. As the ASSR response features we propose pairwise phase-locking-values calculated from the EEG and next classified using binary classifier to detect attended and ignored stimuli. We also report on a possibility to use the stimuli as short as half a second, which is a step forward in ASSR based aBCI. The presented results are helpful for optimization of the aBCI stimuli for…
Universidad de Buenos Aires（布宜诺斯艾利斯大学）；CONICET-Universidad de Buenos Aires. Instituto de Investigación en Ciencias de la Computación (ICC)（CONICET-布宜诺斯艾利斯大学 计算科学研究所（ICC））
Measuring human capabilities to synchronize in time, adapt to perturbations to timing sequences or reproduce time intervals often require experimental setups that allow recording response times with millisecond precision. Most setups present auditory stimuli using either MIDI devices or specialized hardware such as Arduino and are often expensive or require calibration and advanced programming skills. Here, we present in detail an experimental setup that only requires an external sound card and minor electronic skills, works on a conventional PC, is cheaper than alternatives and requires almos…
In this paper, we analyze the cognitive improvements that can be achieved through hacking the brain through the use of multiple methods to enhance neuroplasticity. Exposure to gaming, for example, has proven conducive for learning real-world abilities through auditory and visual stimuli. We will discuss cortical magnification and receptive field sizes, as well as topographic brain maps in the context of neuroplasticity. We finally propose more studies to be performed to improve musical talent and musical abilities.
Using a new probabilistic approach we model the relationship between sequences of auditory stimuli generated by stochastic chains and the electroencephalographic (EEG) data acquired while 19 participants were exposed to those stimuli. The structure of the chains generating the stimuli are characterized by rooted and labeled trees whose leaves, henceforth called contexts, represent the sequences of past stimuli governing the choice of the next stimulus. A classical conjecture claims that the brain assigns probabilistic models to samples of stimuli. If this is true, then the context tree generat…
University of Tsukuba（筑波大学）；RIKEN Brain Science Institute（理化学研究所脑科学研究所）
The paper presents results from a project aiming to create horizontally distributed surround sound sources and virtual sound images as auditory BCI (aBCI) stimuli. The purpose is to create evoked brain wave response patterns depending on attended or ignored sound directions. We propose to use a modified version of the vector based amplitude panning (VBAP) approach to achieve the goal. The so created spatial sound stimulus system for the novel oddball aBCI paradigm allows us to create a multi-command experimental environment with very encouraging results reported in this paper. We also present …
Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Langone School of Medicine, New York University（纽约大学朗格尼医学院）
The superior temporal gyrus (STG) region of cortex critically contributes to speech recognition. In this work, we show that a proposed WaveNet, with limited available data, is able to reconstruct speech stimuli from STG intracranial recordings. We further investigate the impulse response of the fitted model for each recording electrode and observe phoneme level temporospectral tuning properties for the recorded area of cortex. This discovery is consistent with previous studies implicating the posterior STG (pSTG) in a phonetic representation of speech and provides detailed acoustic features th…
In this study we describe a methodology to realize visual images cognition in the broader sense, by a cross-modal stimulation through the auditory channel. An original algorithm of conversion from bi-dimensional images to sounds has been established and tested on several subjects. Our results show that subjects where able to discriminate with a precision of 95\% different sounds corresponding to different test geometric shapes. Moreover, after brief learning sessions on simple images, subjects where able to recognize among a group of 16 complex and never-trained images a single target by heari…
Tactile speech aids, though extensively studied in the 1980s and 90s, never became a commercial success. A hypothesis to explain this failure might be that it is difficult to obtain true perceptual integration of a tactile signal with information from auditory speech: exploitation of tactile cues from a tactile aid might require cognitive effort and so prevent speech understanding at the high rates typical of everyday speech. To test this hypothesis, we attempted to create true perceptual integration of tactile with auditory information in what might be considered the simplest situation encoun…
This paper attempts to address the question that whether the present physical or mathematical theories are sufficient to understand the complexities of human brain when it interacts with the external environment in the form of an auditory stimulus.There have been efforts reporting that the introduction of ambiguity in visual stimuli causes effects which can't be explained classically.In this paper,it is investigated whether ambiguity in auditory stimuli can introduce any non-classical effects in human brain.Simple experiments were performed on normal subjects where they listened to an ambiguou…
We study the extent to which vibrotactile stimuli delivered to the head of a subject can serve as a platform for a brain computer interface (BCI) paradigm. Six head positions are used to evoke combined somatosensory and auditory (via the bone conduction effect) brain responses, in order to define a multimodal tactile and auditory brain computer interface (taBCI). Experimental results of subjects performing online taBCI, using stimuli with a moderately fast inter-stimulus interval (ISI), validate the taBCI paradigm, while the feasibility of the concept is illuminated through information transfe…
This paper reports on a study in which a novel virtual moving sound-based spatial auditory brain-computer interface (BCI) paradigm is developed. Classic auditory BCIs rely on spatially static stimuli, which are often boring and difficult to perceive when subjects have non-uniform spatial hearing perception characteristics. The concept of moving sound proposed and tested in the paper allows for the creation of a P300 oddball paradigm of necessary target and non-target auditory stimuli, which are more interesting and easier to distinguish. We present a report of our study of seven healthy subjec…
Perceptual similarity is a cognitive judgment that represents the end-stage of a complex cascade of hierarchical processing throughout visual cortex. Previous studies have shown a correspondence between the similarity of coarse-scale fMRI activation patterns and the perceived similarity of visual stimuli, suggesting that visual objects that appear similar also share similar underlying patterns of neural activation. Here we explore the temporal relationship between the human brain's time-varying representation of visual patterns and behavioral judgments of perceptual similarity. The visual stim…
Time-reversal symmetry breaking is a key feature of nearly all natural sounds, caused by the physics of sound production. While attention has been paid to the response of the auditory system to "natural stimuli," very few psychophysical tests have been performed. We conduct psychophysical measurements of time-frequency acuity for both "natural" notes (sharp attack, long decay) and time-reversed ones. Our results demonstrate significantly greater precision, arising from enhanced temporal acuity, for such "natural" sounds over both their time-reversed versions and theoretically optimal gaussian …
The spectrotemporal receptive field (STRF) provides a versatile and integrated, spectral and temporal, functional characterization of single cells in primary auditory cortex (AI). In this paper, we explore the origin of, and relationship between, different ways of measuring and analyzing an STRF. We demonstrate that STRFs measured using a spectrotemporally diverse array of broadband stimuli -- such as dynamic ripples, spectrotemporally white noise, and temporally orthogonal ripple combinations (TORCs) -- are very similar, confirming earlier findings that the STRF is a robust linear descriptor …
We investigated, by using auditory models, how three perceptual parameters, loudness, pitch and sharpness, determine human echolocation. We used acoustic recordings from two previous studies, both from stationary situations, and their resulting perceptual data as input to our analysis. An initial analysis was on the room acoustics of the recordings. The parameters of interest were sound pressure level, autocorrelation and spectral centroid. The auditory models were used to analyze echolocation resulting from the perceptual variables, i.e. loudness, pitch and sharpness. Relevant auditory models…
Music listening involves many simultaneous neural operations, including auditory processing, working memory, temporal sequencing, pitch tracking, anticipation, reward, and emotion, and thus, a full investigation of music cognition would benefit from whole-brain analyses. Here, we quantify whole-brain activity while participants listen to a variety of music and speech auditory pieces using two network measures that are grounded in complex systems theory: modularity, which measures the degree to which brain regions are interacting in communities, and flexibility, which measures the rate that bra…
This article questions the widespread assumption that there are brain representations that will always remain unconscious in the sense of being inaccessible to individual awareness under any circumstances. This implies that some part of the knowledge generated by the brain is once and for always excluded from consciousness and, therefore, from being communicated to the outside world. This standpoint neglects the possibility that the human brain might have a capacity for generating metarepresentations of nonconscious knowledge contents at a given moment in time through context sensitive adaptiv…
UCL Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Department of Physics, Blackett Laboratory, Imperial College London（帝国理工学院物理系布莱克特实验室）
Nantes Université（南特大学）；Ecole Centrale Nantes（南特中央理工学院）；CNRS（法国国家科学研究中心）；LS2N（南特数字科学实验室）；Netflix Inc.（网飞公司）
The goal of most subjective studies is to place a set of stimuli on a perceptual scale. This is mostly done directly by rating, e.g. using single or double stimulus methodologies, or indirectly by ranking or pairwise comparison. All these methods estimate the perceptual magnitudes of the stimuli on a scale. However, procedures such as Maximum Likelihood Difference Scaling (MLDS) have shown that considering perceptual distances can bring benefits in terms of discriminatory power, observers' cognitive load, and the number of trials required. One of the disadvantages of the MLDS method is that th…
Early sensory deprivation such as blindness or deafness shapes brain development in multiple ways. While it is established that deprived brain areas start to be engaged in the processing of stimuli from the remaining modalities and in high-level cognitive tasks, some reports have also suggested the possibility of structural and functional changes in non-deprived brain areas. We compared resting-state functional network organization of the brain in early-deaf adults and hearing controls by examining global network segregation and integration. Relative to hearing controls, deaf adults exhibited …
Modelling time-order effects in haptic perception with a Bayesian dynamical framework
Instituto Balseiro, Centro Atómico Bariloche（巴尔塞罗研究所，巴里洛切原子中心）；Department of Applications of Physics and Biology to Health Sciences, Centro Atómico Bariloche（巴里洛切原子中心应用物理与生物健康科学系）；Department of Translational Research, Centro Atómico Bariloche（巴里洛切原子中心转化研究系）；Institute for Intelligent Interactive Systems, Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机学院智能交互系统研究所）
Human similarity judgments are inconsistent with Euclidean, Hamming, Mahalanobis, and the majority of measures used in the extensive literatures on similarity and dissimilarity. From intrinsic properties of brain circuitry, we derive principles of perceptual metrics, showing their conformance to Riemannian geometry. As a demonstration of their utility, the perceptual metrics are shown to outperform JPEG compression. Unlike machine-learning approaches, the outperformance uses no statistics, and no learning. Beyond the incidental application to compression, the metrics offer broad explanatory ac…
Synthesis and Perceptual Scaling of High Resolution Naturalistic Images Using Stable Diffusion
Department of Psychology, Humboldt-Universität zu Berlin（柏林洪堡大学心理学系）；Bernstein Center for Computational Neuroscience Berlin and Berlin Center for Advanced Neuroimaging, Charité Universitätsmedizin Berlin（柏林伯恩斯坦计算神经科学中心和柏林先进神经成像中心，查理医学院）；Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany（马克斯·普朗克人类认知与脑科学研究所，莱比锡，德国）；Max Planck School of Cognition, Leipzig, Germany（马克斯·普朗克认知学校，莱比锡，德国）；Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer’s Disease, Norwegian University of Science and Technology, Trondheim, Norway（凯斯琳系统神经科学研究所，神经计算中心，埃吉和保罗·布雷athen及弗雷德·卡维利微电路中心，朱贝森阿尔茨海默病中心，挪威科学与技术大学，特罗姆瑟，挪威）；Clinic of Neurology, Charité-Universitätsmedizin Berlin, Berlin, Germany（柏林查理医学院神经科诊所）；Berlin Center for Advanced Neuroimaging, Charité-Universitätsmedizin Berlin, Berlin, Germany（柏林先进神经成像中心，查理医学院）；Research Cluster of Excellence “Science of Intelligence”, Technische（卓越研究集群“智能科学”，技术）
Frequency modulation (FM) is a basic constituent of vocalisation in many animals as well as in humans. In human speech, short rising and falling FM-sweeps called formant transitions characterise individual speech sounds. There are two representations of FM in the ascending auditory pathway: a spectral representation, holding the instantaneous frequency of the stimuli; and a sweep representation, consisting of neurons that respond selectively to FM direction. To-date computational models use feedforward mechanisms to explain FM encoding. However, from neuroanatomy we know that there are massive…
