Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.
Large language models struggle with ethnographic text annotation
Calleva Research Centre Oxford Internet Institute University of Oxford(牛津大学奥克斯福德互联网研究所卡列瓦研究中心);Cohn Institute for the History and Philosophy of Science and Ideas Tel Aviv University(特拉维夫大学科恩研究所);Birkbeck College University of London(伦敦大学伯克贝克学院);Centre for the Study of Social Cohesion University of Oxford(牛津大学社会凝聚力研究所以及牛津大学)
Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Use
College of Computing, Georgia Institute of Technology(佐治亚理工学院计算学院);Hospital General Universitario Gregorio Marañón(格雷戈里奥·马拉尼翁大学综合医院);Hofstra University(霍夫斯特拉大学);Columbia University(哥伦比亚大学)
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Maryland(马里兰大学);Virginia Tech(弗吉尼亚理工大学)
Shiny Stories, Hidden Struggles: Investigating the Representation of Disability Through the Lens of LLMs
Department of Foreign Languages and Literature, University of Verona(外国语言文学系,威尼斯大学);Department of Information Engineering and Computer Science, University of Trento(信息工程与计算机科学系,特伦托大学);Data and Web Science Group, University of Mannheim(数据与网络科学小组,曼海姆大学)
Why Do Large Language Models (LLMs) Struggle to Count Letters?
College of Mechanical & Electrical Engineering, Nanjing University of Aeronautics and Astronautics(南京航空航天大学机电学院);ETSI de Telecomunicación, Universidad Politécnica de Madrid(马德里理工大学电信高级工程学院)
Large Language Models Struggle with Unreasonability in Math Problems
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机学院,北京大学);Institute of Artificial Intelligence, Beihang University(人工智能学院,北航);Huawei Noah’s Ark Lab, China(华为诺亚实验室,中国)
Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing
Department of Electrical Engineering, Indian Institute of Technology Delhi(印度理工学院德里电气工程系);Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi(印度理工学院德里人工智能学院)
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It
DISI, University of Trento(特伦托大学DISI中心);MaiNLP, Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心);Free University of Bozen-Bolzano, Italy(博兹纳自由大学)
NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews
University of California, Berkeley(加州大学伯克利分校);University of Southern California(南加州大学);Information Sciences Institute(信息科学研究所);Northeastern University(东北大学)
The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心);Saarland University(萨尔大学)
Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of California, Merced(加州大学默塞德分校);University of Science and Technology of China(中国科学技术大学)