Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly promising area is their ability to interpret code modules from unfamiliar libraries for solving user-instructed tasks. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations. These results raise several open questions: whether demonstrations of library usage is required, whether smaller (and more open) models also possess such capabilities, etc. In this work, we take a broader approach by systematically evaluating a diverse array of LLMs across three scenarios reflecting varying levels of domain specialization to understand their abilities and limitations in generating code based on libraries defined in-context. Our results show that even smaller open-source LLMs like Llama-2 and StarCoder demonstrate an adept understanding of novel code libraries based on specification presented in-context. Our findings further reveal that LLMs exhibit a surprisingly high proficiency in learning novel library modules even when provided with just natural language descriptions or raw code implementations of the functions, which are often cheaper to obtain than demonstrations. Overall, our results pave the way for harnessing LLMs in more adaptable and dynamic coding environments.
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
The Chinese University of Hong Kong(香港中文大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));LightSpeed Studios, Tencent Inc.(腾讯公司光速工作室);Sun Yat-Sen University(中山大学)
SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models
ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University(浙江大学杭州国际科创中心);The Polytechnic Institute, Zhejiang University(浙江大学理工学院);ZJU-UIUC, Zhejiang University(浙江大学伊利诺伊大学厄巴纳-香槟联合学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);AntGroup(蚂蚁集团)
CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);College of Software, Jilin University(吉林大学软件学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.(中国电信人工智能技术(北京)有限公司星辰AGI实验室);Bilibili Inc.(哔哩哔哩公司);University of Ottawa(渥太华大学);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation
College of AI, Tsinghua University(清华大学人工智能学院);Proxseer Inc.(Proxseer公司);School of Computer Science, Nanjing University(南京大学计算机科学与技术学院);Software Institute, Nanjing University(南京大学软件学院);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
University of Cambridge(剑桥大学);Tehran Institute for Advanced Studies (TeIAS)(德黑兰高等研究院(TeIAS));Cardiff University(卡迪夫大学)
通过设计,词嵌入不能够对词的语义的动态性质进行建模,即,词对应于潜在的不同含义的特性。为了解决这一限制,几十个专门的意义表示技术,如意义或上下文嵌入已被提出。然而,尽管这个主题的研究很受欢迎,很少有评估基准存在,专门专注于动态语义的话。在本文中,我们表明,现有的模型已经超过了标准评估数据集的性能上限,即,斯坦福大学的语境词相似度,并指出其不足之处。为了解决缺乏合适的基准测试的问题,我们提出了一个大规模的Word in Context数据集,称为WiC,基于专家策划的注释,用于上下文敏感表示的通用评估。WiC在https://pilehvar.github.io/wic/上发布。
Evaluating Contextualized Language Models for Hungarian
Budapest University of Technology and Economics(布达佩斯技术与经济大学);Institute for Computer Science and Control(计算机科学与控制研究所);Alfréd Rényi Institute of Mathematics(阿尔弗雷德·莱尼数学研究所)
Group of Skills: Group-Structured Skill Retrieval for Agent Skill Libraries
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)科学与工程学院);Sun Yat-sen University(孙中山大学);University of California, San Diego(加州大学圣地亚哥分校);Taiyuan University of Technology(太原理工大学)
HealthContradict: Evaluating Biomedical Knowledge Conflicts in Language Models
Faculty of Medicine, University of Geneva(日内瓦大学医学院);Department of Biomedical Informatics, Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学 Yong Loo Lin 医学院生物医学信息学系);Department of Biostatistics & Bioinformatics, Duke University Artificial Intelligence Institute, National University of Singapore(新加坡国立大学人工智能研究所生物统计学与生物信息学系)