Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. In this study, we experiment with over 30 prompt templates manually written for natural language inference (NLI). We find that models learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively "good" prompts. Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2022). That is, instruction-tuned models often produce good predictions with irrelevant and misleading prompts even at zero shots. In sum, notwithstanding prompt-based models' impressive improvement, we find evidence of serious limitations that question the degree to which such improvement is derived from models understanding task instructions in ways analogous to humans' use of task instructions.
You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments
George Mason University(乔治梅森大学);Virginia Tech(弗吉尼亚理工大学);University of Toronto(多伦多大学);City University of Hong Kong(香港城市大学);Duke Kunshan University(昆山杜克大学)
When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学);LG AI Research(LG人工智能研究院);University of Illinois Chicago(芝加哥大学伊利诺伊分校);University of Michigan(密歇根大学)
Comparative Study on the Discourse Meaning of Chinese and English Media in the Paris Olympics Based on LDA Topic Modeling Technology and LLM Prompt Engineering
Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs
Nara Institute of Science and Technology(奈良科学技術大學);RIKEN(理化学研究所);Guardian Robot Project(守護機器人專案);Kyoto Institute of Technology(京都科技大學);Institute of Science Tokyo(東京科學研究所)
What's the Meaning of Superhuman Performance in Today's NLU?
Babelscape(巴别斯克);Sapienza University of Rome(罗马第一大学);University of Groningen(格罗宁根大学);German Research Center for AI (DFKI)(德国人工智能研究中心);Charles University(查理大学);University of Copenhagen(哥本哈根大学);University of Melbourne(墨尔本大学);Carnegie Mellon University(卡内基梅隆大学);Saarland University(萨尔大学);Jožef Stefan Institute(约泽夫·斯特凡研究所);University of Ljubljana(卢布尔雅那大学);Cardiff University(卡迪夫大学);University of Zurich(苏黎世大学);University of Edinburgh(爱丁堡大学);University of Amsterdam(阿姆斯特丹大学)
Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs
Department of Computer Science and Engineering, College of Engineering, Qatar University(卡塔尔大学计算机科学与工程系);College of Science and Engineering, Hamad Bin Khalifa University (HBKU)(哈马德·本·卡伊夫大学(HBKU)理学院);Primary Health Care Corporation (PHCC)(初级卫生保健公司);Birmingham City University(伯明翰城市大学)