arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.03318cs.CL
收藏

使用自然语言补丁修复模型错误

Fixing Model Bugs with Natural Language Patches

Stanford University(斯坦福大学);Microsoft Research(微软研究院)

Shikhar Murty, Christopher D. Manning, Scott Lundberg, Marco Tulio Ribeiro

摘要:

目前修复NLP模型中系统性问题的方法(例如正则表达式补丁,对更多数据进行微调)要么脆弱,要么劳动密集型,并且容易走捷径。相比之下,人类经常通过自然语言来相互提供更正。从这一点得到灵感,我们探索自然语言补丁-声明性语句,允许开发人员在正确的抽象级别提供纠正反馈,要么覆盖模型("如果评论给2星,情绪是负面的“),要么提供模型可能缺乏的额外信息("如果某个东西被描述为炸弹,那么它是好的”)。我们对确定补丁是否适用的任务与集成补丁信息的任务进行了建模,并表明使用少量的合成数据,我们可以教模型有效地使用真实数据上的真实补丁- 1到7个补丁在情感分析数据集的不同切片上将精度提高1-4个精度点,在关系抽取数据集上将F1提高7个点。最后,我们表明,微调多达100个标记的例子可能需要匹配的性能的一小部分语言补丁。

英文摘要:

Current approaches for fixing systematic problems in NLP models (e.g. regex patches, finetuning on more data) are either brittle, or labor-intensive and liable to shortcuts. In contrast, humans often provide corrections to each other through natural language. Taking inspiration from this, we explore natural language patches -- declarative statements that allow developers to provide corrective feedback at the right level of abstraction, either overriding the model (``if a review gives 2 stars, the sentiment is negative'') or providing additional information the model may lack (``if something is described as the bomb, then it is good''). We model the task of determining if a patch applies separately from the task of integrating patch information, and show that with a small amount of synthetic data, we can teach models to effectively use real patches on real data -- 1 to 7 patches improve accuracy by ~1-4 accuracy points on different slices of a sentiment analysis dataset, and F1 by 7 points on a relation extraction dataset. Finally, we show that finetuning on as many as 100 labeled examples may be needed to match the performance of a small set of language patches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑