With the rapid advancement of Large Language Models (LLMs), the demand for robust instruction-following capabilities in code generation tasks has grown significantly. Code generation not only facilitates faster prototyping and automated testing, but also augments developer efficiency through improved maintainability and reusability of code. In this paper, we introduce CodeIF, the first benchmark specifically designed to assess the abilities of LLMs to adhere to task-oriented instructions within diverse code generation scenarios. CodeIF encompasses a broad range of tasks, including function synthesis, error debugging, algorithmic refactoring, and code explanation, thereby providing a comprehensive suite to evaluate model performance across varying complexity levels and programming domains. We conduct extensive experiments with LLMs, analyzing their strengths and limitations in meeting the demands of these tasks. The experimental results offer valuable insights into how well current models align with human instructions, as well as the extent to which they can generate consistent, maintainable, and contextually relevant code. Our findings not only underscore the critical role that instruction-following LLMs can play in modern software development, but also illuminate pathways for future research aimed at enhancing their adaptability, reliability, and overall effectiveness in automated code generation. CodeIF data and code are publicly available: https://github.com/lin-rany/codeIF
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Trustworthy Autonomous Systems Laboratory at the University of California, Riverside(加州大学河滨分校可信自主系统实验室);University of Michigan(密歇根大学);Workday;University of Southern California(南加州大学);Texas A&M University(德克萨斯A&M大学);Lehigh University(莱斯大学)
Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance
Indiana University Bloomington(印第安纳大学伯明顿分校);Center for Complex Networks and Systems(复杂网络与系统中心)
S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology(深圳先进技术研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Nanjing University(南京大学);Shenzhen Loop Area Institute(深圳河套学院);CentraleSupélec, Université Paris-Saclay(巴黎萨克雷大学CentraleSupélec分校);National University of Singapore(新加坡国立大学)
CodeMEM: AST-Guided Adaptive Memory for Repository-Level Iterative Code Generation
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院)
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Monash University(莫纳什大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61);Singapore Management University(新加坡管理大学);Detomo Inc., Japan(日本Detomo公司);Queen Mary University of London(伦敦玛丽女王大学);University of Notre Dame(圣母大学);TU Darmstadt(达姆施塔特工业大学);Independent(独立研究者);University of Virginia(弗吉尼亚大学);Inria(法国国家信息与自动化研究所);Intel(英特尔公司);Tencent AI Lab(腾讯AI Lab);Cornell University(康奈尔大学);University College London(伦敦大学学院);UNC Chapel Hill(北卡罗来纳大学教堂山分校);UC Berkeley(加州大学伯克利分校);MIT(麻省理工学院);Shanghai Jiaotong University(上海交通大学);Uber(优步公司);UIUC(伊利诺伊大学厄巴纳-香槟分校);Sea AI Lab(Sea AI实验室);AWS AI Labs(亚马逊云科技AI实验室);Contextual AI(Contextual AI公司);Carnegie Mellon University(卡内基梅隆大学);ServiceNow Research(ServiceNow研究院);Hugging Face(Hugging Face公司)
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Hong Kong University of Science and Technology(香港科学与技术大学);DAMO Academy, Alibaba Group(阿里云达摩院)