As language models support larger and larger context sizes, evaluating their ability to make effective use of that context becomes increasingly important. We analyze the ability of several code generation models to handle long range dependencies using a suite of multi-step key retrieval tasks in context windows up to 8k tokens in length. The tasks progressively increase in difficulty and allow more nuanced evaluation of model capabilities than tests like the popular needle-in-the-haystack test. We find that performance degrades significantly for many models (up to 2x) when a function references another function that is defined later in the prompt. We also observe that models that use sliding window attention mechanisms have difficulty handling references further than the size of a single window. We perform simple prompt modifications using call graph information to improve multi-step retrieval performance up to 3x. Our analysis highlights ways that long-context performance needs deeper consideration beyond retrieval of single facts within a document.
AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages
Princeton University(普林斯顿大学);Laboratory for Artificial Intelligence, Princeton University(普林斯顿大学人工智能实验室);Gaston Berger University(加斯顿·伯杰大学);Mila, McGill University(麦吉尔大学米拉研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能教席);Paris Nanterre University(巴黎南泰尔大学);Paris-Saclay University(巴黎-萨克雷大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);LORIA(洛林计算机科学实验室);Université de Lorraine(洛林大学);University of Trento(特伦托大学);University of Minnesota–Twin Cities(明尼苏达大学双城分校);Imperial College London(伦敦帝国学院);Binghamton University(宾汉姆顿大学);Makerere University(马凯雷雷大学);Penn State University(宾夕法尼亚州立大学);Mbarara University of Science and Technology(姆巴拉拉科技大学);Chalmers University of Technology(查尔姆斯理工大学);University of Ibadan(伊巴丹大学);Nnamdi Azikiwe University(纳姆迪·阿齐基韦大学);South African Centre for Digital Language Resources(南非数字语言资源中心)
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation
Peter Munk Cardiac Centre, University Health Network(彼得·蒙克心脏中心,大学健康网络);University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(向量人工智能研究所)
What Makes Agent Memory Useful for Reliable Unanswerable Question Handling?
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Suzhou City University(苏州城市学院);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学技术与人工智能学院);Shanghai Innovation Institute(上海创新研究院)
Mic Drop or Data Flop? Evaluating the Fitness for Purpose of AI Voice Interviewers for Data Collection within Quantitative & Qualitative Research Contexts
CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps
Department of Artificial Intelligence, Chung-Ang University(Chung-Ang大学人工智能系);Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University(Chung-Ang大学高级成像科学研究院,多媒体与电影)
PII-Bench: Evaluating Query-Aware Privacy Protection Systems
Institute of Fintech, Fudan University(复旦大学金融科技学院);Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(复旦大学数据科学上海重点实验室);Laboratory of Data Analytics and Security, Fudan University(复旦大学数据安全分析实验室)
State Key Lab of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);Key Lab of AI Safety, Chinese Academy of Sciences(人工智能安全重点实验室,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions
Shanghai University of Finance and Economics(上海财经大学);University of Edinburgh(爱丁堡大学);Southern University of Science and Technology(南方科技大学);Fudan University(复旦大学)
HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs
Kyushu University(九州大学);hon9kon9ize;The Education University of Hong Kong(香港教育大学);The University of Hong Kong(香港大学);Independent Researcher(独立研究者);Votee AI;Beever AI
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Fudan University(复旦大学);NJU-LINK Team, Nanjing University(南京大学NJU-LINK团队);University of Science and Technology of China(中国科学技术大学);Kuaishou Technology(快手科技)