Large language models (LLMs) are typically aligned to a universal set of safety and usage principles intended for broad public acceptability. Yet, real-world applications of LLMs often take place within organizational ecosystems shaped by distinctive corporate policies, regulatory requirements, use cases, brand guidelines, and ethical commitments. This reality highlights the need for rigorous and comprehensive evaluation of LLMs with pluralistic alignment goals, an alignment paradigm that emphasizes adaptability to diverse user values and needs. In this work, we present PLURALISTIC BEHAVIOR SUITE (PBSUITE), a dynamic evaluation suite designed to systematically assess LLMs' capacity to adhere to pluralistic alignment specifications in multi-turn, interactive conversations. PBSUITE consists of (1) a diverse dataset of 300 realistic LLM behavioral policies, grounded in 30 industries; and (2) a dynamic evaluation framework for stress-testing model compliance with custom behavioral specifications under adversarial conditions. Using PBSUITE, We find that leading open- and closed-source LLMs maintain robust adherence to behavioral policies in single-turn settings (less than 4% failure rates), but their compliance weakens substantially in multi-turn adversarial interactions (up to 84% failure rates). These findings highlight that existing model alignment and safety moderation methods fall short in coherently enforcing pluralistic behavioral policies in real-world LLM interactions. Our work contributes both the dataset and analytical framework to support future research toward robust and context-aware pluralistic alignment techniques.
Offscript: Automated Auditing of Instruction Adherence in LLMs
University of Washington Information School Seattle Washington USA；University of Washington\ G. Allen School of Computer Science \& Engineering Seattle Washington USA；University of Washington Information School；University of Washington\ G. Allen School of Computer Science \& Engineering
Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values
Microsoft Research Asia（微软亚洲研究院）；Fudan University（复旦大学）；Tsinghua University（清华大学）；The University of Chicago（芝加哥大学）；Renmin University of China（中国人民大学）；William & Mary（威廉与玛丽大学）
Instructional Goal-Aligned Question Generation for Student Evaluation in Virtual Lab Settings: How Closely Do LLMs Actually Align?
Department of EdPsych, University of Wisconsin-Madison（威斯康星大学麦迪逊分校教育心理学系）；Department of CS, Wichita State University（威斯康星州立大学Wichita分校计算机科学系）；Department of CSSE, Auburn University（阿伯茨罕大学计算机科学与工程系）
Customizing Open Source LLMs for Quantitative Medication Attribute Extraction across Heterogeneous EHR Systems
Department of Statistics, UC Riverside（统计学系，加州大学河滨分校）；Department of Electrical and Computer Engineering, UCLA（电气与计算机工程系，加州大学洛杉矶分校）；Department of Psychiatry and Biobehavioral Sciences, UCLA（精神病学与生物行为科学系，加州大学洛杉矶分校）
SAGE: A Service Agent Graph-guided Evaluation Benchmark
Tianjin University（天津大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；The Chinese University of Hong Kong（香港中文大学）；Independent Researcher（独立研究员）
