Large language models (LLMs) are known to exhibit brittle behavior under adversarial prompts and jailbreak attacks, even after extensive alignment and fine-tuning. This fragility reflects a broader challenge of modern neural language models: small, carefully structured perturbations in high-dimensional input spaces can induce large and unpredictable changes in internal semantic representations and output.
We investigate monotonicity as an architectural inductive bias for improving the robustness of Transformer-based language models. Monotonicity constrains semantic transformations so that strengthening information, evidence, or constraints cannot lead to regressions in the corresponding internal representations. Such order-preserving behavior has long been exploited in control and safety-critical systems to simplify reasoning and improve robustness, but has traditionally been viewed as incompatible with the expressivity required by neural language models.
We show that this trade-off is not inherent. By enforcing monotonicity selectively in the feed-forward sublayers of sequence-to-sequence Transformers -- while leaving attention mechanisms unconstrained -- we obtain monotone language models that preserve the performance of their pretrained counterparts. This architectural separation allows negation, contradiction, and contextual interactions to be introduced explicitly through attention, while ensuring that subsequent semantic refinement is order-preserving. Empirically, monotonicity substantially improves robustness: adversarial attack success rates drop from approximately 69% to 19%, while standard summarization performance degrades only marginally.
Structural Disentanglement in Bilinear MLPs via Architectural Inductive Bias
Department of Electrical Engineering, Indian Institute of Technology Roorkee (IIT Roorkee), India；Department of Material Science \& Engineering, IIT Roorkee, India；Mehta Family School of Data Science
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
Mila & Université de Montréal（Mila与蒙特利尔大学）；Noah’s Ark Lab（Noah’s Ark实验室）；Advanced Micro Devices；Chandar Research Lab（Chandar研究实验室）；Polytechnique Montréal（蒙特利尔理工学院）；CIFAR AI Chair（CIFAR人工智能主席）
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
Laboratoire de Physique de l’Ecole normale supérieure ENS（巴黎高等师范学院物理实验室）；Université PSL（巴黎文理研究大学）；CNRS（法国国家科学研究中心）；Sorbonne Université（索邦大学）；Université Paris-Diderot（巴黎第七大学）；Sorbonne Paris Cité（巴黎索邦西岱大学）；Facebook AI Research（脸书人工智能研究院）；Facebook（脸书）；Courant Institute of Mathematical Sciences and Center for Data Science（库朗数学科学研究所与数据科学中心）；New York University（纽约大学）
Simulating a Bias Mitigation Scenario in Large Language Models
Department of Computer Science, Bowling Green State University（计算机科学系，布恩维尔州立大学）；School of Computing, University of Nebraska–Lincoln（计算学院，内布拉斯加-林肯大学）；Department of Civil, Environmental, and Ocean Engineering, Stevens Institute of Technology（土木、环境与海洋工程系，史蒂文斯理工学院）；Department of Computer Science, University of South Carolina（计算机科学系，南卡罗来纳大学）；Artificial Intelligence Institute, University of South Carolina（人工智能研究院，南卡罗来纳大学）
From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling
Department of Computer Science, National Center for Text Mining, The University of Manchester（计算机科学系，文本挖掘国家中心，曼彻斯特大学）；Department of Computer Science and Engineering, Islamic University of Technology（计算机科学与工程系，伊斯兰技术大学）
Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan（国家台湾大学计算机科学与信息工程系）；Institute of Information Science, Academia Sinica, Taiwan（台湾学术院信息科学研究所）；AI Research Center (AINTU), National Taiwan University, Taiwan（国家台湾大学人工智能研究中心（AINTU））
