通过语句微调在编码器模型上实现自然零样本提示
Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning
AI总结：
本文提出Statement-Tuning技术，通过将判别任务建模为有限语句判别来训练编码器模型，以较少参数实现与大型语言模型竞争的零样本和少样本跨任务泛化性能。
摘要：
虽然大型语言模型（LLMs）在零样本和少样本场景中展现出卓越的能力，但它们通常需要计算上难以承受的规模。相反，像BERT和RoBERTa这样较小的掩码语言模型（MLMs）通过微调取得了最先进的结果，但由于其架构限制，难以扩展到少样本和零样本设置。因此，我们提出了Statement-Tuning，一种将判别任务建模为一组有限语句并训练编码器模型在潜在语句之间进行判别以确定标签的技术。我们在多个任务上进行Statement-Tuning以实现跨任务泛化。实验结果表明，Statement-Tuning以显著更少的参数实现了与最先进LLMs相比具有竞争力的性能。此外，该研究调查了若干设计选择对少样本和零样本泛化的影响，揭示Statement-Tuning可以在适度的训练数据下实现强大性能，并受益于任务和语句多样性以提升对未见任务的泛化能力。
英文摘要：
While Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot scenarios, they often require computationally prohibitive sizes. Conversely, smaller Masked Language Models (MLMs) like BERT and RoBERTa achieve state-of-the-art results through fine-tuning but struggle with extending to few-shot and zero-shot settings due to their architectural constraints. Hence, we propose Statement-Tuning, a technique that models discriminative tasks as a set of finite statements and trains an encoder model to discriminate between the potential statements to determine the label. We do Statement-Tuning on multiple tasks to enable cross-task generalization. Experimental results demonstrate that Statement-Tuning achieves competitive performance compared to state-of-the-art LLMs with significantly fewer parameters. Moreover, the study investigates the impact of several design choices on few-shot and zero-shot generalization, revealing that Statement-Tuning can achieve strong performance with modest training data and benefits from task and statement diversity for unseen task generalizability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
