LINGO：视觉化消除自然语言指令偏差以支持任务多样性
LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity
AI总结：
本文提出可视化分析界面 LINGO，通过识别、修改和评估自然语言任务指令中的偏差，帮助用户创建语言多样性更高、指令偏差更低的任务，并在涵盖 55 种语言、1,616 个任务的数据集上验证其有效性。
摘要：
跨任务泛化是定义自然语言理解掌握程度的重要成果。人类在这方面表现出非凡的能力，只要给出文本指令形式的任务定义和少量示例，就能解决许多不同类型的任务。近期关于预训练语言模型的工作模仿了这种学习方式：用户可以通过一系列自然语言提示或指令来定义并示例一个任务，让模型尝试完成。虽然与传统的监督学习相比，提示方法带来了更高的跨任务泛化能力，但分析提供给模型的任务指令中的“偏差”是一个困难的问题，因此相对较少被探索。例如，我们真正建模的是任务本身，还是建模了用户的指令？为了帮助研究这一问题，我们开发了 LINGO，一种新颖的可视化分析界面，支持有效的、任务驱动的工作流程，以（1）帮助识别自然语言任务指令中的偏差，（2）修改（或创建）任务指令以减少偏差，以及（3）评估预训练模型在去偏任务指令上的性能。为了稳健地评估 LINGO，我们对一个包含 1,616 个语言任务及其自然语言指令、涵盖 55 种不同语言的数据集，开展了一项包含新手和专家指令创建者的用户研究。对于两组用户，LINGO 都促进了为预训练模型创建更具难度的任务，这些任务包含更高的语言多样性和更低的指令偏差。我们还讨论了在开发和评估 LINGO 过程中获得的见解如何有助于未来仪表板的设计，这些仪表板旨在最小化跨多个领域创建提示所涉及的工作量。
英文摘要：
Cross-task generalization is a significant outcome that defines mastery in natural language understanding. Humans show a remarkable aptitude for this, and can solve many different types of tasks, given definitions in the form of textual instructions and a small set of examples. Recent work with pre-trained language models mimics this learning style: users can define and exemplify a task for the model to attempt as a series of natural language prompts or instructions. While prompting approaches have led to higher cross-task generalization compared to traditional supervised learning, analyzing 'bias' in the task instructions given to the model is a difficult problem, and has thus been relatively unexplored. For instance, are we truly modeling a task, or are we modeling a user's instructions? To help investigate this, we develop LINGO, a novel visual analytics interface that supports an effective, task-driven workflow to (1) help identify bias in natural language task instructions, (2) alter (or create) task instructions to reduce bias, and (3) evaluate pre-trained model performance on debiased task instructions. To robustly evaluate LINGO, we conduct a user study with both novice and expert instruction creators, over a dataset of 1,616 linguistic tasks and their natural language instructions, spanning 55 different languages. For both user groups, LINGO promotes the creation of more difficult tasks for pre-trained models, that contain higher linguistic diversity and lower instruction bias. We additionally discuss how the insights learned in developing and evaluating LINGO can aid in the design of future dashboards that aim to minimize the effort involved in prompt creation across multiple domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
