It is now common practice in software development for large language models (LLMs) to be used to generate program code. It is desirable to evaluate the robustness of LLMs for this usage. This paper is concerned in particular with how sensitive LLMs are to variations in descriptions of the coding tasks. However, existing techniques for evaluating this robustness are unsuitable for code generation because the input data space of natural language descriptions is discrete. To address this problem, we propose a robustness evaluation method called scenario domain analysis, which aims to find the expected minimal change in the natural language descriptions of coding tasks that would cause the LLMs to produce incorrect outputs. We have formally proved the theoretical properties of the method and also conducted extensive experiments to evaluate the robustness of four state-of-the-art art LLMs: Gemini-pro, Codex, Llamma2 and Falcon 7B, and have found that we are able to rank these with confidence from best to worst. Moreover, we have also studied how robustness varies in different scenarios, including the variations with the topic of the coding task and with the complexity of its sample solution, and found that robustness is lower for more complex tasks and also lower for more advanced topics, such as multi-threading and data structures.
Operationalising the Right to be Forgotten in LLMs: A Lightweight Sequential Unlearning Framework for Privacy-Aligned Deployment in Politically Sensitive Environments
Department of Mathematics(数学系);Munster Technological University(穆恩斯特技术大学);Department of Computer Science(计算机科学系)
BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems
University of Graz, Graz, Austria(格拉茨大学);Supervised Program for Alignment Research (SPAR)(对齐研究监督计划 (SPAR));Centre pour la Sécurité de l'IA (CeSIA), Paris, France(人工智能安全研究中心 (CeSIA),巴黎,法国)
LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs
Electronics and Telecommunications Research Institute(电子电信研究院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Hugging Face
FD-LLM: Large Language Model for Fault Diagnosis of Machines
School of Mechatronic Engineering(机电工程学院);China University of Mining and Technology(中国矿业大学);School of Computer Science and Technology(计算机科学与技术学院);School of Software Engineering(软件工程学院);Sun Yat-Sen University(中山大学);Institute of Data Science(数据科学研究院);National University of Singapore(新加坡国立大学)
Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
Georgia Institute of Technology(佐治亚理工学院);University of Bristol(布里斯托大学);University of Maryland(马里兰大学);University College London(伦敦大学学院);MATS;Astra;MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)
Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling
Ubiquitous Knowledge Processing Lab (UKP Lab)(通用知识处理实验室);Department of Computer Science(计算机科学系);Hessian Center for AI (hessian.AI)(黑森人工智能中心);Technical University of Darmstadt(达姆施塔特技术大学)
Harnessing non-adversarial robustness in large language models
Applied AI Institute, Moscow, Russia(莫斯科应用人工智能研究所);King's College London, London, UK(伦敦国王学院);International Joint Laboratory of AI for Industry, QUST, Qingdao, China(工业人工智能联合实验室)
Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics
School of Computer Sciences(计算机科学学院);National Institute of Science Education and Research(国家科学教育与研究 institute);An OCC of Homi Bhabha National Institute, India(霍米·巴布国家研究所办公室,印度)
Survey and Evaluation of Converging Architecture in LLMs based on Footsteps of Operations
Department of Electrical and Electronic Engineering(电子工程系);Department of Systems Semiconductor Engineering(半导体系统工程系);BK21 Graduate Program in Intelligent Semiconductor Technology(智能半导体技术研究生项目)
ObfusQAte: A Proposed Framework to Evaluate LLM Robustness on Obfuscated Factual Question Answering
Indian Institute of Technology Patna(印度帕纳布理工大学);Manipal University Jaipur(贾伊普尔曼尼帕尔大学);Indian Institute of Science Education and Research Kolkata(印度科学教育与研究学院科利卡塔)
Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments
Multi-Stage Retrieval for Operational Technology Cybersecurity Compliance Using Large Language Models: A Railway Casestudy
Digital Transit Limited, 3M Buckley Innovation Centre(数字交通有限公司,3M Buckley创新中心);Department of Computer Science at University of Huddersfield(赫德瑟尔大学计算机科学系)
RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学上海智能信息处理重点实验室)