Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty.
Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of California, Santa Barbara(加州大学圣巴巴拉分校);Beijing Wenge Technology Co., Ltd(北京文格科技有限公司)
Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching
Departamento de Computación, FCEyN, Universidad de Buenos Aires(布宜诺斯艾利斯大学计算机系);Instituto de Ciencias de la Computación, FCEyN, Universidad de Buenos Aires(布宜诺斯艾利斯大学计算机科学研究所);Lumina Labs(Lumina实验室)
Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints
National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Tsinghua University(清华大学)
Reason to Rote: Rethinking Memorization in Reasoning
Department of ICS, Utrecht University(信息科学与计算系,乌特勒支大学);MaiNLP, Center for Information and Language Processing, LMU Munich(MaiNLP,信息与语言处理中心,慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Saarland Informatics Campus, Saarland University(萨尔兰州信息科学校区,萨尔兰州大学)
Rethinking On-policy Optimization for Query Augmentation
University of Utah(犹他大学);The University of Queensland(昆士兰大学);University of Waterloo(滑铁卢大学);New York University(纽约大学);University of Notre Dame(圣母大学);Université de Montréal(蒙特利尔大学);Google DeepMind(谷歌DeepMind);University of Oklahoma(俄克拉荷马大学)
Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations
Graduate Institute of Communication Engineering, National Taiwan University(台湾大学通信工程研究院);Computer Science Department, University of California, Los Angeles(加州大学洛杉矶分校计算机科学系);Graduate Institute of Networking and Multimedia, National Taiwan University(台湾大学网络与多媒体研究院);Department of Electrical Engineering, National Taiwan University(台湾大学电子工程系)
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Ping An Technology (Shenzhen) Co., Ltd.(平安科技(深圳)有限公司);University of Pittsburgh(匹兹堡大学);University of Maryland, College Park(马里兰大学学院公园分校);University of Connecticut(康涅狄格大学)
From Instruction to Output: The Role of Prompting in Modern NLG
Faculty of Information Technology, Monash University(莫纳什大学信息科技学院);School of Computer, Data and Mathematical Sciences, Western Sydney University(西澳大学计算机、数据与数学科学学院);School of Computing, Macquarie University(麦考瑞大学计算学院)
Rethinking the Chain-of-Thought: The Roles of In-Context Learning and Pre-trained Priors
School of Intelligence Science and Technology, National Key Laboratory for Novel Software Technology, Nanjing University(智能科学与技术学院,新型软件技术国家重点实验室,南京大学);School of Computing and Information Systems, Singapore Management University(计算与信息系统学院,新加坡管理大学);Central South University(中南大学);School of Global Education and Development, International Chinese Language Education, University of Chinese Academy of Social Sciences(全球教育与发展学院,国际中文教育,中国社会科学院)
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
Stanford University(斯坦福大学);Shenzhen University(深圳大学);The Chinese University of Hong Kong(香港中文大学);City University of Hong Kong(香港城市大学);Renmin University of China(中国人民大学)