Large language models underestimate the impact of negations on how much they change the meaning of a sentence. Therefore, learned evaluation metrics based on these models are insensitive to negations. In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric. For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset. Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity. Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.
Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电子与计算机工程系);Amazon(亚马逊);Samsung Research(三星研究院);School of Computer Science and Engineering, Soongsil University(顺天大学计算机科学与工程学院);IPAI, AIIS, ASRI, INMC, and ISRC, Seoul National University(首尔国立大学IPAI、AIIS、ASRI、INMC和ISRC)
The Sound of Absence: Audio-Language Embedding Models Struggle with Negation
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan(国家台湾大学通讯工程研究所);Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan(国家台湾大学人工智能卓越研究中心)
Negation Neglect: When models fail to learn negations in training
University of Oxford(牛津大学);University of Toronto(多伦多大学);Warsaw University of Technology(华沙技术大学);NASK National Research Institute(国家研究 institute NASK);Truthful AI;Anthropic;UC Berkeley(伯克利大学)
Negation Blindness in Large Language Models: Unveiling the NO Syndrome in Image Generation
Aligarh Muslim University(阿里格尔穆斯林大学);VIT Bhopal University(VIT博帕尔大学);Nanyang Technological University(南洋理工大学);Technical University of Munich(慕尼黑工业大学);Imperial College London(伦敦帝国学院);Edinburgh Napier University(爱丁堡龙比亚大学)
Context-Aware Classification and Grading of Sensitive Information in Online Conversational Health Data
School of Computing, Macquarie University, Australia(麦考瑞大学计算机学院);Department of Biomedical Engineering, National University of Singapore(新加坡国立大学生物医学工程系);School of Mathematics and Statistics, Shandong University of Technology, China(山东理工大学数学与统计学院);Digital Intelligence Center, Fuzhou University Affiliated Provincial Hospital, China(福州市附属省医院数字智能中心);School of Computer Science and Engineering, UNSW, Australia(新南威尔士大学计算机科学与工程学院)
ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoning
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);National University of Singapore(新加坡国立大学);Chinese University of Hong Kong(香港中文大学);University of Oxford(牛津大学);Tsinghua University(清华大学)
CSRP: Chain-of-Thought Reasoning for Chinese Text Correction via Reinforcement Learning with Efficiency-Aware Rewards
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(东华大学教育人工智能研究所)