Recent work shows Large Language Models (LLMs) struggle to understand natural language constraints for various text generation tasks in zero- and few-shot settings. While, in the code domain, there is wide usage of constraints in code format to maintain the integrity of code written in Domain-Specific Languages (DSLs) like JSON and YAML which are widely used for system-level programming tasks in enterprises. Given that LLMs are increasingly used for system-level code tasks, evaluating if they can comprehend these code constraints is crucial. However, no work has been done to evaluate their controllability over code constraints. Hence, we introduce ConCodeEval, a first-of-its-kind benchmark having two novel tasks for code constraints across five representations. Our findings suggest that language models struggle with code constraints. Code languages that perform excellently for normal code tasks do not perform well when the same languages represent fine-grained constraints.
SpecEval: Evaluating Model Adherence to Behavior Specifications
Department of Computer Science, Stanford University（计算机科学系，斯坦福大学）；The Bradley Department of Electrical and Computer Engineering, Virginia Tech（电气与计算机工程系，弗吉尼亚理工学院）
YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese
Kobe University（神户大学）；National Institute of Advanced Industrial Science and Technology (AIST)（国立研究开发法人产业技术综合研究所（AIST））；The University of Tokyo（东京大学）；RIKEN（理化学研究所）；Tohoku University（东北大学）
OpenHuEval: Evaluating Large Language Model on Hungarian Specifics
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；HUN-REN Hungarian Research Centre for Linguistics（匈牙利语言研究与研究中心）；Wuhan University（武汉大学）；Shanghai University（上海大学）；University of Chinese Academy of Sciences（中国科学院大学）；East China Normal University（华东师范大学）；Peking University（北京大学）；Harbin Institute of Technology（哈尔滨工业大学）；Tsinghua University（清华大学）；Chinese University of Hong Kong（香港大学）
Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所，图宾根）；ETH Zurich, Zurich（苏黎世联邦理工学院，苏黎世）；University of Tübingen, Methods Center, Tübingen（图宾根大学，方法中心，图宾根）；Massachusetts Institute of Technology, Cambridge（麻省理工学院，剑桥）
CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns
Nanyang Technological University（南洋理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Huazhong University of Science and Technology（华中科技大学）
GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning
MemTensor (Shanghai) Technology Co., Ltd.（MemTensor（上海）科技有限公司）；University of Sydney（悉尼大学）；Research Institute of China Telecom（中国电信研究院）；Renmin University of China（中国人民大学）
VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models
Institute of BigData, Fudan University（复旦大学大数据研究院）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；School of Computer Science, Huazhong University of Science and Technology（华中科技大学计算机科学学院）；Purple Mountain Laboratories（紫金山实验室）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件学院）
From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
Department of Computer Science, King Khalid University（国王 Khalid 大学计算机科学系）；Department of Computer Science and Artificial Intelligence, Umm Al-Qura University（乌姆·阿勒·库拉大学计算机科学与人工智能系）；Imam Abdulrahman Bin Faisal University（伊玛目阿卜杜勒拉赫曼·本·法伊萨尔大学）
Evaluating Cognitive Age Alignment in Interactive AI Agents
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Shenzhen Children's Hospital（深圳儿童医院）；Peking University（北京大学）；Hong Kong Polytechnic University（香港理工大学）
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
Department of Data Science & AI, Monash University（数据科学与人工智能系，莫纳什大学）；Faculty of Law, Monash University（法学院，莫纳什大学）
Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai University（上海大学）
Are We Evaluating the Edit Locality of LLM Model Editing Properly?
National University of Singapore（新加坡国立大学）；Huazhong University of Science（华中科技大学）；Central China Normal University；University of Maryland, College Park（马里兰大学学院市分校）
SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems
Nanjing University（南京大学）；Microsoft Research Asia（微软亚洲研究院）；University of British Columbia（不列颠哥伦比亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Evaluating the performance of state-of-the-art esg domain-specific pre-trained large language models in text classification against existing models and traditional machine learning techniques
Microsoft Research Asia（微软亚洲研究院）；Beijing Normal University（北京师范大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；ValgrAI；The Psychometrics Centre, University of Cambridge（剑桥大学心理测量中心）；Cambridge Judge Business School（剑桥大学嘉治商学院）
Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Universidad Politécnica de Madrid（马德里理工大学）；Universidad de Valladolid（巴利亚多利德大学）；Universidad de Salamanca（萨拉曼卡大学）；Ghent University（根特大学）
From Online User Feedback to Requirements: Evaluating Large Language Models for Classification and Specification Tasks
Lero, the Research Ireland Centre for Software, University of Limerick（利默里克大学软件研究爱尔兰研究中心 Lero）；University College Dublin (UCD)（都柏林大学学院）；University of Padova（帕多瓦大学）
