Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles
Indian Institute of Technology Kharagpur(印度理工学院Khargapur分校);Rochester Institute of Technology(罗切斯特理工学院);Researcher, Fatima Fellowship(研究员,Fatima fellowship);AI Institute, University of South Carolina(人工智能研究所,南卡罗来纳大学);Amazon GenAI(亚马逊生成人工智能);Stanford University(斯坦福大学);James Silberrad Brown Center for AI, San Diego State University(詹姆斯·西伯拉德·布朗人工智能中心,圣地亚哥州立大学)
Assessing the effectiveness of large language models (LLMs) in performing different tasks is crucial for understanding their strengths and weaknesses. This paper presents Hierarchical Prompting Taxonomy (HPT), grounded on human cognitive principles and designed to assess LLMs by examining the cognitive demands of various tasks. The HPT utilizes the Hierarchical Prompting Framework (HPF), which structures five unique prompting strategies in a hierarchical order based on their cognitive requirement on LLMs when compared to human mental capabilities. It assesses the complexity of tasks with the Hierarchical Prompting Index (HPI), which demonstrates the cognitive competencies of LLMs across diverse datasets and offers insights into the cognitive demands that datasets place on different LLMs. This approach enables a comprehensive evaluation of an LLMs problem solving abilities and the intricacy of a dataset, offering a standardized metric for task complexity. Extensive experiments with multiple datasets and LLMs show that HPF enhances LLM performance by 2% to 63% compared to baseline performance, with GSM8k being the most cognitively complex task among reasoning and coding tasks with an average HPI of 3.20 confirming the effectiveness of HPT. To support future research and reproducibility in this domain, the implementations of HPT and HPF are available here.
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
Universidade de Vigo, Department of Computer Science, ESEI-Higher School of Computer Engineering(维戈大学计算机科学系ESEI高等计算机工程学院);IFCAE-Institute for Research in Physics, Computing and Aerospace Science(IFCAE物理、计算与航空航天科学研究所)
Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes
Mayo Clinic College of Medicine and Science(梅奥诊所医学与科学学院);School of Computing and AI, Arizona State University(亚利桑那州立大学计算与人工智能学院);Mayo Clinic Comprehensive Cancer Center(梅奥诊所综合癌症中心);Department of Radiation Oncology, Mayo Clinic(梅奥诊所放射肿瘤科);Department of Oncology, Mayo Clinic(梅奥诊所肿瘤科);Department of Artificial Intelligence and Informatics, Mayo Clinic(梅奥诊所人工智能与信息学系);Dana-Farber Cancer Institute, Harvard Medical School(哈佛医学院丹娜-法伯癌症研究所)
The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
University of Maryland(马里兰大学);Learn Prompting;OpenAI;Stanford(斯坦福大学);Microsoft(微软公司);Vanderbilt(范德堡大学);Princeton(普林斯顿大学);Texas State University(德克萨斯州立大学);Icahn School of Medicine(伊坎医学院);ASST Brianza(布里安扎综合医院);Mount Sinai Beth Israel(西奈山 Beth Israel 医院);Instituto de Telecomunicações(电信研究所);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)
PHAnToM: Persona-based Prompting Has An Effect on Theory-of-Mind Reasoning in Large Language Models
Institute of Data Science(数据科学研究所);NUS Centre for Trusted Internet and Community, National University of Singapore(新加坡国立大学可信互联网与社区中心);Amazon(亚马逊);Stanford University(斯坦福大学);Tsinghua University(清华大学)
From Instruction to Output: The Role of Prompting in Modern NLG
Faculty of Information Technology, Monash University(莫纳什大学信息科技学院);School of Computer, Data and Mathematical Sciences, Western Sydney University(西澳大学计算机、数据与数学科学学院);School of Computing, Macquarie University(麦考瑞大学计算学院)
Cognitive-Mental-LLM: Evaluating Reasoning in Large Language Models for Mental Health Prediction via Online Text
Ira A. Fulton Schools of Engineering Arizona State University Tempe, USA(伊拉·A·福林工程学校亚利桑那州立大学Tempe分校);Department of Psychology Santa Clara University Santa Clara, USA(心理学系圣克拉拉大学Santa Clara分校)
Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situations
Seoul National University(首尔国立大学);Human-Computer Interaction Institute, Carnegie Mellon University(人机交互研究所,卡内基梅隆大学);Language Technologies Institute, Carnegie Mellon University(语言技术研究所,卡内基梅隆大学)
Analyzing Cognitive Differences Among Large Language Models through the Lens of Social Worldview
Wangxuan Institute of Computer Technology, Peking University(北京大学计算机技术研究院);Information Management Department, Peking University(北京大学信息管理系);Renmin University of China(中国人民大学)
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
Department of Electronic Engineering, Tsinghua University, Beijing, China(清华大学电子工程系);College of AI, Tsinghua University, Beijing, China(清华大学人工智能学院);School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China(北京邮电大学计算机科学学院);iFLYTEK Research, Hefei, China(iFLYTEK研究院)
FoodTaxo: Generating Food Taxonomies with Large Language Models
ADAPT Centre, School of Computing, Dublin City University(ADAPT中心、计算学院、都柏林城市大学);Lucerne School of Computer Science and Information Technology (HSLU)(卢塞恩计算机科学与信息科技学院(HSLU))
Retrieval-Augmented Guardrails for AI-Drafted Patient-Portal Messages: Error Taxonomy Construction and Large-Scale Evaluation
Stanford Center for Biomedical Informatics Research(斯坦福大学生物医学信息学研究中心);Department of Medicine, Stanford University(斯坦福大学医学院);Department of Biomedical Data Science, Stanford University(斯坦福大学生物医学数据科学系);Stanford Clinical Excellence Research Center, Stanford University(斯坦福大学临床卓越研究中心)
From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions
City University of Hong Kong(香港城市大学);Zhejiang Normal University(浙江师范大学);Squirrel Ai Learning;University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学)
BloomWise: Enhancing Problem-Solving capabilities of Large Language Models using Bloom's-Taxonomy-Inspired Prompts
Speech and Language Processing Group, National Technical University of Athens(国家技术大学雅典语音与语言处理组);Institute for Language and Speech Processing, Athena Research Center(雅典研究中心语言与语音处理研究所);National Technical University of Athens(国家技术大学雅典)
Visual Large Language Models Exhibit Human-Level Cognitive Flexibility in the Wisconsin Card Sorting Test
Laboratory of Brain Atlas and Brain-inspired Intelligence, Institute of Automation, Chinese Academy of Sciences(脑图谱与脑启发智能实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Inria centre at the university of Bordeaux, Neurodegenerative Diseases Institute(波尔多大学Inria中心,神经退行性疾病研究所)