Recent work has demonstrated that pre-trained language models (PLMs) are zero-shot learners. However, most existing zero-shot methods involve heavy human engineering or complicated self-training pipelines, hindering their application to new situations. In this work, we show that zero-shot text classification can be improved simply by clustering texts in the embedding spaces of PLMs. Specifically, we fit the unlabeled texts with a Bayesian Gaussian Mixture Model after initializing cluster positions and shapes using class names. Despite its simplicity, this approach achieves superior or comparable performance on both topic and sentiment classification datasets and outperforms prior works significantly on unbalanced datasets. We further explore the applicability of our clustering approach by evaluating it on 14 datasets with more diverse topics, text lengths, and numbers of classes. Our approach achieves an average of 20% absolute improvement over prompt-based zero-shot learning. Finally, we compare different PLM embedding spaces and find that texts are well-clustered by topics even if the PLM is not explicitly pre-trained to generate meaningful sentence embeddings. This work indicates that PLM embeddings can categorize texts without task-specific fine-tuning, thus providing a new way to analyze and utilize their knowledge and zero-shot learning ability.
Bidirectional Language Models Are Also Few-shot Learners
University of Pennsylvania（宾夕法尼亚大学）；Microsoft（微软）；Google Research（谷歌研究院）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）
大语言模型，例如GPT-3（Brown等人，2020）可以执行任意任务，而无需在仅使用几个标记示例进行提示后进行微调。一个任意的任务可以被重新表述为一个自然语言提示，一个语言模型可以被要求生成完成，间接地执行任务的范式称为基于语义的学习。到目前为止，基于突现的学习能力主要是针对单向语言模型来证明的。然而，在去噪目标上预训练的双向语言模型（例如掩蔽语言建模）为迁移学习产生更强的学习表示。这激发了促进双向模式的可能性，但它们的预培训目标使它们在很大程度上与现有的激励范式不相容。我们提出了SAP（顺序自回归拟合），一种技术，使双向模型的提示。利用机器翻译任务作为案例研究，我们提出了双向mT5模型（Xue et al.，2021）与SAP进行比较，并证明其Few-Shot和zero-shot转换优于单向模型（如GPT-3和XGLM）的Few-Shot转换（Lin et al.，2021），尽管mT5的参数减少了约50%。我们进一步表明SAP在问答和摘要方面是有效的。这是第一次，我们的研究结果表明，基于语义的学习是一个更广泛的一类语言模型，而不仅仅是单向模型的新兴属性。
Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines
Università degli Studi di Milano Bicocca（米兰比可卡大学）；Universitat Pompeu Fabra（庞培法布拉大学）；University of Regensburg（雷根斯堡大学）；Politecnico di Milano（米兰理工大学）；Istituto per le Tecnologie Didattiche (ITD-CNR)（国家研究委员会教育技术研究所）；Università degli Studi di Milano（米兰大学）；Institute of Science and Technology of Cognition (ISTC-CNR)（国家研究委员会认知科学与技术研究所）
Use Me Wisely: AI-Driven Assessment for LLM Prompting Skills Development
Università degli Studi di Milano Bicocca（米兰比可卡大学）；University of Regensburg（雷根斯堡大学）；Universitat Pompeu Fabra（庞培法布拉大学）；Politecnico di Milano（米兰理工大学）；National Research Council of Italy (CNR)（意大利国家研究委员会）
Causal Prompting for Implicit Sentiment Analysis with Large Language Models
School of Computing Technologies, RMIT University（RMIT大学计算机技术学院）；Institute of Innovation, Science and Sustainability, Federation University Australia（联邦大学澳大利亚创新、科学与可持续性研究所）；CSIRO’s Data61（CSIRO数据61）
Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiaotong University（上海交通大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）
Clustering as Reasoning: A $k$-Means Interpretation of Chain-of-Thought Graph Learning
University of Electronic Science and Technology of China（电子科技大学）；Singapore Management University（新加坡国立大学）；Michigan State University（密歇根州立大学）；The Chinese University of Hong Kong（香港中文大学）
Transition Network Analysis: A Novel Framework for Modeling, Visualizing, and Identifying the Temporal Patterns of Learners and Learning Processes
University of Eastern Finland（东芬兰大学）；University of Oulu（奥卢大学）；University of Oslo（奥斯陆大学）；FernUniversität in Hagen（哈根函授大学）；University of Bergen（卑尔根大学）；University of Jyväskylä（于韦斯屈莱大学）
