FinchGPT:一种用于鸟鸣分析的基于Transformer的语言模型
FinchGPT: a Transformer based language model for birdsong analysis
AI总结:
本研究开发了基于Transformer的FinchGPT模型用于鸟鸣分析,通过在文本化鸟鸣语料库上训练,有效捕捉音节序列中的长程依赖,并揭示了计算与生物学操作对模型输出的影响,为破译动物发声复杂性和探索非人类交流系统提供了新框架。
摘要:
源于层级结构的token间长程依赖是人类语言的一个显著特征。然而,非人类动物的序列发声中是否存在类似的依赖关系仍是一个研究课题。以建模token间长程依赖能力而闻名的Transformer架构,为研究这一现象提供了强大工具。在本研究中,我们采用Transformer架构分析了孟加拉雀(Lonchura striata domestica)的鸣声,其特点是音节序列高度可变且复杂。为此,我们开发了FinchGPT,这是一种基于Transformer的模型,在鸟鸣文本化语料库上进行训练,在该领域的表现优于其他架构模型。注意力权重分析表明,FinchGPT能有效捕捉音节序列内的长程依赖。此外,逆向工程方法展示了计算和生物学操作对其性能的影响:限制FinchGPT的注意力跨度,以及通过消融特定脑核团破坏鸟鸣句法,均显著影响了模型输出。我们的研究凸显了大语言模型(LLMs)在破译动物发声复杂性方面的变革潜力,为探索非人类交流系统的结构特性提供了新框架,同时揭示了生物大脑与人工神经网络之间的计算差异。
英文摘要:
The long-range dependencies among the tokens, which originate from hierarchical structures, are a defining hallmark of human language. However, whether similar dependencies exist within the sequential vocalization of non-human animals remains a topic of investigation. Transformer architectures, known for their ability to model long-range dependencies among tokens, provide a powerful tool for investigating this phenomenon. In this study, we employed the Transformer architecture to analyze the songs of Bengalese finch (Lonchura striata domestica), which are characterized by their highly variable and complex syllable sequences. To this end, we developed FinchGPT, a Transformer-based model trained on a textualized corpus of birdsongs, which outperformed other architecture models in this domain. Attention weight analysis revealed that FinchGPT effectively captures long-range dependencies within syllables sequences. Furthermore, reverse engineering approaches demonstrated the impact of computational and biological manipulations on its performance: restricting FinchGPT's attention span and disrupting birdsong syntax through the ablation of specific brain nuclei markedly influenced the model's outputs. Our study highlights the transformative potential of large language models (LLMs) in deciphering the complexities of animal vocalizations, offering a novel framework for exploring the structural properties of non-human communication systems while shedding light on the computational distinctions between biological brains and artificial neural networks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。