The long-context capabilities of large language models (LLMs) have been a hot topic in recent years. To evaluate the performance of LLMs in different scenarios, various assessment benchmarks have emerged. However, as most of these benchmarks focus on identifying key information to answer questions, which mainly requires the retrieval ability of LLMs, these benchmarks can partially represent the reasoning performance of LLMs from large amounts of information. Meanwhile, although LLMs often claim to have context windows of 32k, 128k, 200k, or even longer, these benchmarks fail to reveal the actual supported length of these LLMs. To address these issues, we propose the LongIns benchmark dataset, a challenging long-context instruction-based exam for LLMs, which is built based on the existing instruction datasets. Specifically, in our LongIns, we introduce three evaluation settings: Global Instruction & Single Task (GIST), Local Instruction & Single Task (LIST), and Local Instruction & Multiple Tasks (LIMT). Based on LongIns, we perform comprehensive evaluations on existing LLMs and have the following important findings: (1). The top-performing GPT-4 with 128k context length performs poorly on the evaluation context window of 16k in our LongIns. (2). For the multi-hop reasoning ability of many existing LLMs, significant efforts are still needed under short context windows (less than 4k).
PACIT: Unlocking the Power of Examples for Better In-Context Instruction Tuning
Nanjing University(南京大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Southern University of Science and Technology(南方科技大学);Shanghai University of Finance and Economics(上海财经大学)
University of Southern California(南加州大学);University of Chicago(芝加哥大学);Temple University(天普大学);University of Birmingham(伯明翰大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究院);Penn State University(宾夕法尼亚州立大学)
Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of California, Santa Cruz(加州大学圣克鲁兹分校);Hong Kong Baptist University(香港 Baptist 大学)
One-Shot Learning as Instruction Data Prospector for Large Language Models
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);The University of Sydney(悉尼大学)
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);School of Computer Science, Peking University(北京大学计算机学院);PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications(北京大学-CMCC(湖北)大语言模型工业应用联合实验室);School of Health Sciences and Biomedical Engineering, Hebei University of Technology(河北工业大学健康科学与生物医学工程学院)
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Microsoft(微软);Singapore University of Technology and Design(新加坡科技设计大学);Peking University(北京大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学)
LongForm: Effective Instruction Tuning with Reverse Instructions
Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning(慕尼黑机器学习中心);Language Technology Lab, University of Cambridge(剑桥大学语言技术实验室)
IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages
Nilekani Centre at AI4Bharat(AI4Bharat的Nilekani中心);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);IT University of Copenhagen(丹麦技术大学);Microsoft(微软)
Strategy-Induct: Task-Level Strategy Induction for Instruction Generation
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan(国立台湾大学计算机科学与资讯工程学系);Institute of Information Science, Academia Sinica, Taiwan(台湾“中央研究院”资讯科学研究所);AI Research Center (AINTU), National Taiwan University, Taiwan(国立台湾大学人工智能研究中心)
Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究所)
BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing
Manning College of Information and Computer Sciences, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校曼宁信息与计算机科学学院);University of Massachusetts Medical School(马萨诸塞大学医学院);Miner School of Computer and Information Sciences, University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校迈纳计算机与信息科学学院);VA Bedford Health Care(贝德福德退伍军人保健中心)