Large Language Models have taken the cognitive science world by storm. It is perhaps timely now to take stock of the various research paradigms that have been used to make scientific inferences about ``cognition" in these models or about human cognition. We review several emerging research paradigms -- GPT-ology, LLMs-as-computational-models, and ``silicon sampling" -- and review recent papers that have used LLMs under these paradigms. In doing so, we discuss their claims as well as challenges to scientific inference under these various paradigms. We highlight several outstanding issues about LLMs that have to be addressed to push our science forward: closed-source vs open-sourced models; (the lack of visibility of) training data; and reproducibility in LLM research, including forming conventions on new task ``hyperparameters" like instructions and prompts.
Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization
University of California, San Diego（加利福尼亚大学圣迭戈分校）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）
Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering
University of California San Francisco（加州大学旧金山分校）；Waymark
SPL: A Socratic Playground for Learning Powered by Large Language Model
Institute for Intelligent Systems, University of Memphis（孟菲斯大学智能系统研究所）；University of Memphis（孟菲斯大学）；School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）；School of Psychology, Shaanxi Normal University（陕西师范大学心理学院）；School of Psychology, Central China Normal University（华中师范大学心理学院）；Department of Applied Social Sciences, Hong Kong Polytechnic University（香港理工大学应用社会科学系）
Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation
IDEA Research, International Digital Economy Academy（国际数字经济学院IDEA研究院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology（香港科技大学）
MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
University of California, Los Angeles Computer Science Department（加州大学洛杉矶分校计算机科学系）；University of Pennsylvania Department of Materials Science and Engineering（宾夕法尼亚大学材料科学与工程系）；Virginia Tech Department of Computer Science（弗吉尼亚理工大学计算机科学系）
GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT
School of Geography and Information Engineering, China University of Geosciences（中国地质大学地理与信息工程学院）；National Engineering Research Center for Geographic Information System, China University of Geosciences（中国地质大学地理信息系统国家工程研究中心）
Evaluating Frontier LLMs on PhD-Level Mathematical Reasoning: A Benchmark on a Textbook in Theoretical Computer Science about Randomized Algorithms
Wyoming Seminary（怀俄明神学院）；San Jose State University（圣何塞州立大学）；Guilin University of Electronic Technology（桂林电子科技大学）；University of California, Berkeley（加州大学伯克利分校）；Northeastern University（东北大学）；Guangdong University of Technology（广东工业大学）
ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Centre for Medical Informatics, The University of Edinburgh（爱丁堡大学医学信息学中心）；Health Data Research UK（英国健康数据研究所）；School of Computer Science, Peking University（北京大学计算机学院）；National Engineering Research Center for Software Engineering, Peking University（北京大学软件工程国家工程研究中心）；ETH Zurich（苏黎世联邦理工学院）；School of Computing and Data Science, The University of Hong Kong（香港大学计算与数据科学学院）
Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognition
Department of Construction Management, Colorado State University（科罗拉多州立大学建设管理系）；Department of Civil, Construction, and Environmental Engineering, North Carolina State University（北卡罗来纳州立大学土木、建设与环境工程系）
