arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.12644cs.CLcs.AI
收藏

分层提示分类法:一种与人类认知原则对齐的大语言模型通用评估框架

Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles

Indian Institute of Technology Kharagpur(印度理工学院Khargapur分校);Rochester Institute of Technology(罗切斯特理工学院);Researcher, Fatima Fellowship(研究员,Fatima fellowship);AI Institute, University of South Carolina(人工智能研究所,南卡罗来纳大学);Amazon GenAI(亚马逊生成人工智能);Stanford University(斯坦福大学);James Silberrad Brown Center for AI, San Diego State University(詹姆斯·西伯拉德·布朗人工智能中心,圣地亚哥州立大学)

Devichand Budagam, Ashutosh Kumar, Mahsa Khoshnoodi, Sankalp KJ, Vinija Jain, Aman Chadha

AI总结:

本文提出基于人类认知原则的分层提示分类法HPT及分层提示框架HPF,通过分层提示指数HPI评估任务复杂度和LLM认知能力,实验显示性能提升2%至63%。

摘要:

评估大语言模型(LLMs)在执行不同任务时的有效性对于理解其优势与不足至关重要。本文提出了分层提示分类法(Hierarchical Prompting Taxonomy,HPT),该方法以人类认知原则为基础,旨在通过考察各类任务的认知需求来评估大语言模型。HPT采用分层提示框架(Hierarchical Prompting Framework,HPF),该框架根据与人类心智能力相比时对大语言模型提出的认知要求,将五种独特的提示策略按层级顺序进行结构化组织。HPT利用分层提示指数(Hierarchical Prompting Index,HPI)评估任务复杂度,该指数展示了大语言模型在不同数据集上的认知能力,并揭示了不同数据集对不同大语言模型所提出的认知需求。这一方法能够全面评估大语言模型的问题解决能力以及数据集的复杂程度,为任务复杂度提供了一个标准化度量指标。在多个数据集和大语言模型上进行的大量实验表明,与基线性能相比,HPF将大语言模型的性能提升了2%至63%,其中GSM8k是推理和编码任务中认知复杂度最高的任务,其平均HPI为3.20,这证实了HPT的有效性。为支持该领域未来的研究和可复现性,HPT和HPF的实现已在此处公开提供。

英文摘要:

Assessing the effectiveness of large language models (LLMs) in performing different tasks is crucial for understanding their strengths and weaknesses. This paper presents Hierarchical Prompting Taxonomy (HPT), grounded on human cognitive principles and designed to assess LLMs by examining the cognitive demands of various tasks. The HPT utilizes the Hierarchical Prompting Framework (HPF), which structures five unique prompting strategies in a hierarchical order based on their cognitive requirement on LLMs when compared to human mental capabilities. It assesses the complexity of tasks with the Hierarchical Prompting Index (HPI), which demonstrates the cognitive competencies of LLMs across diverse datasets and offers insights into the cognitive demands that datasets place on different LLMs. This approach enables a comprehensive evaluation of an LLMs problem solving abilities and the intricacy of a dataset, offering a standardized metric for task complexity. Extensive experiments with multiple datasets and LLMs show that HPF enhances LLM performance by 2% to 63% compared to baseline performance, with GSM8k being the most cognitively complex task among reasoning and coding tasks with an average HPI of 3.20 confirming the effectiveness of HPT. To support future research and reproducibility in this domain, the implementations of HPT and HPF are available here.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑