This study investigates the internal mechanisms of BERT, a transformer-based large language model, with a focus on its ability to cluster narrative content and authorial style across its layers. Using a dataset of narratives developed via GPT-4, featuring diverse semantic content and stylistic variations, we analyze BERT's layerwise activations to uncover patterns of localized neural processing. Through dimensionality reduction techniques such as Principal Component Analysis (PCA) and Multidimensional Scaling (MDS), we reveal that BERT exhibits strong clustering based on narrative content in its later layers, with progressively compact and distinct clusters. While strong stylistic clustering might occur when narratives are rephrased into different text types (e.g., fables, sci-fi, kids' stories), minimal clustering is observed for authorial style specific to individual writers. These findings highlight BERT's prioritization of semantic content over stylistic features, offering insights into its representational capabilities and processing hierarchy. This study contributes to understanding how transformer models like BERT encode linguistic information, paving the way for future interdisciplinary research in artificial intelligence and cognitive neuroscience.
Pre-trained language models, such as BERT, have achieved significant accuracy gain in many natural language processing tasks. Despite its effectiveness, the huge number of parameters makes training a BERT model computationally very challenging. In this paper, we propose an efficient multi-stage layerwise training (MSLT) approach to reduce the training time of BERT. We decompose the whole training process into several stages. The training is started from a small model with only a few encoder layers and we gradually increase the depth of the model by adding new encoder layers. At each stage, we …
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Unity Health Toronto（多伦多联合健康）
Transformer language models have shown remarkable ability in detecting when a word is anomalous in context, but likelihood scores offer no information about the cause of the anomaly. In this work, we use Gaussian models for density estimation at intermediate layers of three language models (BERT, RoBERTa, and XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark. In lower layers, surprisal is highly correlated to low token frequency, but this correlation diminishes in upper layers. Next, we gather datasets of morphosyntactic, semantic, and commonsense anomalies from ps…
Domain-Adapted Pre-trained Language Models for Implicit Information Extraction in Crash Narratives
Department of Technology, Management and Economics, Technical University of Denmark（技术、管理与经济系，丹麦技术大学）；Department of Mechanics and Maritime Sciences, Chalmers University of Technology（机械与航海科学系，查尔姆斯理工大学）；Department of Computer Science and Engineering, Chalmers University of Technology（计算机科学与工程系，查尔姆斯理工大学）
How LLMs Build Fictional Worlds: Setting and Narrative Space in AI-Generated Creative Storytelling
FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；LMU München（慕尼黑大学）；Institute of AI for Health, Helmholtz Zentrum München（慕尼黑亥姆霍兹中心人工智能健康研究所）；University of Birmingham（伯明翰大学）
Identifying Symptoms of Delirium from Clinical Narratives Using Natural Language Processing
University of Florida（佛罗里达大学）；UF Health Shands Hospital（UF Health Shands医院）；University of California Los Angeles（加州大学洛杉矶分校）；UF Clinical and Translational Science Institute（UF临床与转化科学研究所）；North Florida/South Georgia Veterans Health Service（北佛罗里达/南乔治亚退伍军人卫生服务处）；College of Nursing, University of Florida（佛罗里达大学护理学院）；College of Public Health and Health Professions & College of Medicine, University of Florida（佛罗里达大学公共卫生与健康专业学院及医学院）
Several pre-training objectives, such as masked language modeling (MLM), have been proposed to pre-train language models (e.g. BERT) with the aim of learning better language representations. However, to the best of our knowledge, no previous work so far has investigated how different pre-training objectives affect what BERT learns about linguistics properties. We hypothesize that linguistically motivated objectives such as MLM should help BERT to acquire better linguistic knowledge compared to other non-linguistically motivated objectives that are not intuitive or hard for humans to guess the …
Detecting HIV-Related Stigma in Clinical Narratives Using Large Language Models
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida（佛罗里达大学医学院健康结果与生物医学信息学系）；Department of Epidemiology, College of Public Health and Health Professions, University of Florida（佛罗里达大学公共卫生与健康专业学院流行病学系）；Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida（佛罗里达大学莉莉安·S·威尔斯神经外科系普雷斯顿·A·威尔斯脑肿瘤治疗中心）
Google Brain（谷歌大脑）；Google（谷歌）；UC Berkeley（加州大学伯克利分校）；UCLA（加州大学洛杉矶分校）
Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning rates trains ResNet on ImageNet in a few minutes. However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks. In this paper, we first study a principled layerwise adaptation strategy to accelerate training of …
Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida（健康结局与生物医学信息学系，佛罗里达大学医学院）；Division of Cardiovascular Medicine, Department of Medicine, College of Medicine, University of Florida（心血管医学部，医学院，佛罗里达大学）；Division of Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida（血液学与肿瘤学部，医学院，佛罗里达大学）
CACER: Clinical Concept Annotations for Cancer Events and Relations
University of Washington（华盛顿大学）；George Mason University（乔治梅森大学）；Huntsman Cancer Institute（亨茨曼癌症研究所）；University of Utah（犹他大学）；Virginia Commonwealth University（弗吉尼亚联邦大学）
Large-scale transformer models have become the de-facto architectures for various machine learning applications, e.g., CV and NLP. However, those large models also introduce prohibitive training costs. To mitigate this issue, we propose a novel random and layerwise token dropping method (random-LTD), which skips the computation of a subset of the input tokens at all middle layers. Particularly, random-LTD achieves considerable speedups and comparable accuracy as the standard training baseline. Compared to other token dropping methods, random-LTD does not require (1) any importance score-based …
Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA)（达莱莫勒人工智能研究所）
We present two deep learning approaches to narrative text understanding for character relationship modelling. The temporal evolution of these relations is described by dynamic word embeddings, that are designed to learn semantic changes over time. An empirical analysis of the corresponding character trajectories shows that such approaches are effective in depicting dynamic evolution. A supervised learning approach based on the state-of-the-art transformer model BERT is used instead to detect static relations between characters. The empirical validation shows that such events (e.g., two charact…
Detecting the Clinical Features of Difficult-to-Treat Depression using Synthetic Data from Large Language Models
University of Oxford（牛津大学）；University of Liverpool（利物浦大学）；Mersey Care NHS Foundation Trust（默西护理国民保健信托基金会）；Oxford Health NHS Foundation Trust（牛津健康国民保健信托基金会）；NIHR Oxford Health Biomedical Research Centre（NIHR牛津健康生物医学研究中心）
University of Florida（佛罗里达大学）；Weill Cornell Medicine（威尔康奈尔医学院）
Social and behavioral determinants of health (SBDoH) have important roles in shaping people's health. In clinical research studies, especially comparative effectiveness studies, failure to adjust for SBDoH factors will potentially cause confounding issues and misclassification errors in either statistical analyses and machine learning-based models. However, there are limited studies to examine SBDoH factors in clinical outcomes due to the lack of structured SBDoH information in current electronic health record (EHR) systems, while much of the SBDoH information is documented in clinical narrati…
Glasgow Caledonian University（格拉斯哥卡里多尼亚大学）；University of the West of Scotland（西苏格兰大学）；Lay Summaries Ltd（Lay Summaries 有限公司）；University of Edinburgh（爱丁堡大学）
Summarisation of research results in plain language is crucial for promoting public understanding of research findings. The use of Natural Language Processing to generate lay summaries has the potential to relieve researchers' workload and bridge the gap between science and society. The aim of this narrative literature review is to describe and compare the different text summarisation approaches used to generate lay summaries. We searched the databases Web of Science, Google Scholar, IEEE Xplore, Association for Computing Machinery Digital Library and arXiv for articles published until 6 May 2…
Motivation: A perennial challenge for biomedical researchers and clinical practitioners is to stay abreast with the rapid growth of publications and medical notes. Natural language processing (NLP) has emerged as a promising direction for taming information overload. In particular, large neural language models facilitate transfer learning by pretraining on unlabeled text, as exemplified by the successes of BERT models in various NLP applications. However, fine-tuning such models for an end task remains challenging, especially with small labeled datasets, which are common in biomedical NLP. Res…
