大语言模型(LLMs)的许多安全和对齐失败源于分布外(OOD)情境:模型开发者未预见到的异常提示或响应模式。我们通过引入名为Misalignment Out Of Distribution (MOOD)的基准,系统研究LLM监控流程能否检测这些OOD对齐失败。对于在大量安全数据集上训练的现成模型,很难找到真正OOD的失败。我们通过在MOOD中包含一个受限训练集(用于训练我们自己的监控器)以及七个具有不同对齐失败且超出训练分布的测试集来规避这一问题。利用MOOD,我们发现守卫模型(安全分类器)通常难以泛化到OOD。为解决此问题,我们提出将守卫模型与OOD检测器结合。我们测试了四种OOD检测器,发现将守卫模型与基于马氏距离和困惑度的OOD检测器结合,可将召回率从39%提升至45%。我们还建立了跨模型规模的监控器(结合守卫模型和OOD检测器)的正向扩展趋势;发现将OOD检测纳入监控比使用参数多20倍的守卫模型能获得更高的召回率增益。我们的工作表明,OOD检测应成为LLM监控的关键组成部分,并为这一重要问题的进一步研究奠定了基础。我们公开发布了实验代码和数据,相关链接见:https://github.com/Dylan102938/mood-bench。
英文摘要:
Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by model developers. We systematically study whether LLM monitoring pipelines can detect these OOD alignment failures by introducing a benchmark called Misalignment Out Of Distribution (MOOD). It is difficult to find failures that are truly OOD for off-the-shelf models trained on vast safety datasets. We sidestep this by including a restricted training set in MOOD that we use to train our own monitors, as well as seven test sets with diverse alignment failures that are outside the training distribution. Using MOOD, we find that guard models (safety classifiers) often fail to generalize OOD. To fix this, we propose combining guard models with OOD detectors. We test four types of OOD detectors and find that a combination of a guard model with Mahalanobis distance and perplexity-based OOD detectors can improve recall from 39% to 45%. We also establish positive scaling trends across model scales for monitors that combine a guard model and OOD detector; we find that incorporating OOD detection into monitoring achieves a higher recall gain than using a guard model with 20 times more parameters. Our work suggests that OOD detection should be a crucial component of LLM monitoring and provides a foundation for further work on this important problem.
We release the code and data for our experiments publicly, and you can find the relevant links here: https://github.com/Dylan102938/mood-bench.
PreAct-Bench: Benchmarking Predictive Monitoring in LLMs
King’s College London(伦敦国王学院);National University of Singapore(新加坡国立大学);Southern University of Science and Technology(南方科技大学);Thomson Reuters Foundational Research(汤姆森路透基础研究);Imperial College London(伦敦帝国学院);The Alan Turing Institute(艾伦·图灵研究所)
HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring
School of Computing and Information Systems, University of Melbourne, Australia(墨尔本大学计算机与信息系统学院);School of Computer Science, University of Auckland, New Zealand(奥克兰大学计算机科学学院)
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
School of Software, Beihang University(北京航空航天大学软件学院);School of Computer Science, Peking University(北京大学计算机学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Galbot(盖尔博特(Galbot))
An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Department of Computer Science University of Illinois Urbana–Champaign(计算机科学系伊利诺伊大学厄巴纳-香槟分校);Department of Computer Science Stanford University(计算机科学系斯坦福大学)
Benchmarking LLM Judges for Mobile Agent Evaluation
Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Concordia University(康考迪亚大学);University of Toronto(多伦多大学);Shanghai University(上海大学);McMaster University(麦克马斯特大学)
GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures
AI-NLP-ML Research Laboratory(人工智能-自然语言处理-机器学习研究实验室);Department of Computer Science and Engineering, Indian Institute of Technology Patna(计算机科学与工程系,印度理工学院帕坦)
Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agents
City University of Hong Kong(香港城市大学);ByteDance(字节跳动);Yale University(耶鲁大学);Fudan University(复旦大学);Hong Kong Baptist University(香港浸会大学);Dalian University of Technology(大连理工大学)
GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theory
ETH Zürich(苏黎世联邦理工学院);Berea College(贝雷学院);University of Toronto(多伦多大学);Vector Institute(向量研究所);Max Planck Institute for Intelligent Systems, Tübingen, Germany(图宾根德国智能系统马克斯·普朗克研究所)