使用自然语言补丁修复模型错误
Fixing Model Bugs with Natural Language Patches
摘要:
目前修复NLP模型中系统性问题的方法(例如正则表达式补丁,对更多数据进行微调)要么脆弱,要么劳动密集型,并且容易走捷径。相比之下,人类经常通过自然语言来相互提供更正。从这一点得到灵感,我们探索自然语言补丁-声明性语句,允许开发人员在正确的抽象级别提供纠正反馈,要么覆盖模型("如果评论给2星,情绪是负面的“),要么提供模型可能缺乏的额外信息("如果某个东西被描述为炸弹,那么它是好的”)。我们对确定补丁是否适用的任务与集成补丁信息的任务进行了建模,并表明使用少量的合成数据,我们可以教模型有效地使用真实数据上的真实补丁- 1到7个补丁在情感分析数据集的不同切片上将精度提高1-4个精度点,在关系抽取数据集上将F1提高7个点。最后,我们表明,微调多达100个标记的例子可能需要匹配的性能的一小部分语言补丁。
英文摘要:
Current approaches for fixing systematic problems in NLP models (e.g. regex patches, finetuning on more data) are either brittle, or labor-intensive and liable to shortcuts. In contrast, humans often provide corrections to each other through natural language. Taking inspiration from this, we explore natural language patches -- declarative statements that allow developers to provide corrective feedback at the right level of abstraction, either overriding the model (``if a review gives 2 stars, the sentiment is negative'') or providing additional information the model may lack (``if something is described as the bomb, then it is good''). We model the task of determining if a patch applies separately from the task of integrating patch information, and show that with a small amount of synthetic data, we can teach models to effectively use real patches on real data -- 1 to 7 patches improve accuracy by ~1-4 accuracy points on different slices of a sentiment analysis dataset, and F1 by 7 points on a relation extraction dataset. Finally, we show that finetuning on as many as 100 labeled examples may be needed to match the performance of a small set of language patches.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。