Large language models (LLMs) show remarkable capabilities across a variety of tasks. Despite the models only seeing text in training, several recent studies suggest that LLM representations implicitly capture aspects of the underlying grounded concepts. Here, we explore LLM representations of a particularly salient kind of grounded knowledge -- spatial relationships. We design natural-language navigation tasks and evaluate the ability of LLMs, in particular GPT-3.5-turbo, GPT-4, and Llama2 series models, to represent and reason about spatial structures. These tasks reveal substantial variability in LLM performance across different spatial structures, including square, hexagonal, and triangular grids, rings, and trees. In extensive error analysis, we find that LLMs' mistakes reflect both spatial and non-spatial factors. These findings suggest that LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains.
ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models
Peking University(北京大学);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Department of Sociology, Peking University(北京大学社会学系);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院)
Evaluating Semantic and Syntactic Understanding in Large Language Models for Payroll Systems
Center for Entity Resolution(实体解析中心);Information Quality (ERIQ) - University of Arkansas - Little Rock, Arkansas, Little Rock, USA(信息质量(ERIQ)- 阿拉巴马大学-利特尔岩石分校,阿肯色州,利特尔岩石,美国)
VCIFBench: Evaluating Complex Instruction Following for Video Understanding
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(知识驱动人机智能工程研究中心,吉林大学);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works
School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(复旦大学计算机科学技术学院上海市数据科学重点实验室)
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海教育人工智能研究院)
SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学蚂蚁集团知识图谱联合实验室);Ant Group(蚂蚁集团)
YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese
Kobe University(神户大学);National Institute of Advanced Industrial Science and Technology (AIST)(国立研究开发法人产业技术综合研究所(AIST));The University of Tokyo(东京大学);RIKEN(理化学研究所);Tohoku University(东北大学)
Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学教育部计算语言学重点实验室)
SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models
ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University(浙江大学杭州国际科创中心);The Polytechnic Institute, Zhejiang University(浙江大学理工学院);ZJU-UIUC, Zhejiang University(浙江大学伊利诺伊大学厄巴纳-香槟联合学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);AntGroup(蚂蚁集团)
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
Westlake University(西湖大学);Westlake Institute for Advanced Study(西湖高等研究院);Central South University(中南大学);University of Electronic Science and Technology of China(电子科技大学);Microsoft Research Asia(微软亚洲研究院)
HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs
Kyushu University(九州大学);hon9kon9ize;The Education University of Hong Kong(香港教育大学);The University of Hong Kong(香港大学);Independent Researcher(独立研究者);Votee AI;Beever AI
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
Department of Informatics(信息学院);King’s College London(伦敦国王学院);Institute of Psychology(心理学研究所);University of Tartu(塔尔图大学);Department of Psychology(心理学系);Stanford University(斯坦福大学);The Alan Turing Institute(艾伦·图灵研究所)
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
NLP;CT Lab, Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门大学CT实验室);Department of Portuguese, Faculty of Arts and Humanities, University of Macau(人文学院葡萄牙语系,澳门大学)