Large Language Models (LLMs) are susceptible to security and safety threats, such as prompt injection, prompt extraction, and harmful requests. One major cause of these vulnerabilities is the lack of an instruction hierarchy. Modern LLM architectures treat all inputs equally, failing to distinguish between and prioritize various types of instructions, such as system messages, user prompts, and data. As a result, lower-priority user prompts may override more critical system instructions, including safety protocols. Existing approaches to achieving instruction hierarchy, such as delimiters and instruction-based training, do not address this issue at the architectural level. We introduce the Instructional Segment Embedding (ISE) technique, inspired by BERT, to modern large language models, which embeds instruction priority information directly into the model. This approach enables models to explicitly differentiate and prioritize various instruction types, significantly improving safety against malicious prompts that attempt to override priority rules. Our experiments on the Structured Query and Instruction Hierarchy benchmarks demonstrate an average robust accuracy increase of up to 15.75% and 18.68%, respectively. Furthermore, we observe an improvement in instruction-following capability of up to 4.1% evaluated on AlpacaEval. Overall, our approach offers a promising direction for enhancing the safety and effectiveness of LLM architectures.
LLMs can be easily Confused by Instructional Distractions
IPAI, Seoul National University(IPAI,首尔国立大学);LG AI Research(LG人工智能研究所);Dept. of ECE, Seoul National University(电子工程系,首尔国立大学);SNU-LG AI Research Center(SNU-LG人工智能研究所)
Overview of the NLPCC 2025 Shared Task 4: Multi-modal, Multilingual, and Multi-hop Medical Instructional Video Question Answering Challenge
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究所,中国科学院);School of Computer Science and Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学);School of Engineering, Westlake University(工程学院,西湖大学)
Instructional Prompt Optimization for Few-Shot LLM-Based Recommendations on Cold-Start Users
Cullen College of Engineering(卡伦工程学院);McKelvey School of Engineering(麦凯维工程学院);University of Houston(休斯顿大学);Washington University in St. Louis(圣路易斯华盛顿大学);Arizona State University(亚利桑那州立大学);Indiana University Bloomington(印第安纳大学伯明顿分校)