arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.12868cs.CL
收藏

基于大型预训练语言模型的提示式语义解析的鲁棒性:Codex上的实证研究

On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex

CSIRO’s Data61(澳大利亚联邦科学与工业研究组织数据61研究所);Monash University(莫纳什大学)

Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, Yuan-Fang Li

AI总结:

本文针对大型提示式代码语言模型Codex的对抗鲁棒性开展首次实证研究,发现SOTA代码语言模型易受精心构造的对抗样本攻击,并提出无需大量标注数据或计算资源即可提升鲁棒性的方法。

摘要:

语义解析是一种旨在构建自然语言问题含义的结构化表示的技术。近期,在代码上训练的少样本语言模型的进展表明,与在下游任务上训练的传统单模态语言模型相比,这些模型在生成此类表示时表现出更优的性能。尽管取得了这些进展,现有的微调神经语义解析器仍易受到针对自然语言输入的对抗攻击。虽然已有研究证实,通过对抗训练可提升小型语义解析器的鲁棒性,但该方法在实际场景中不适用于大型语言模型,因为它需要大量计算资源以及对领域内语义解析数据的昂贵人工标注。本文针对大型提示式代码语言模型Codex的对抗鲁棒性开展了首次实证研究。研究结果表明,当前最先进(SOTA)的代码语言模型易受到精心构造的对抗样本的攻击。为应对这一挑战,本文提出了无需大量标注数据或大量计算资源即可提升鲁棒性的方法。

英文摘要:

Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advancements in few-shot language models trained on code have demonstrated superior performance in generating these representations compared to traditional unimodal language models, which are trained on downstream tasks. Despite these advancements, existing fine-tuned neural semantic parsers are susceptible to adversarial attacks on natural-language inputs. While it has been established that the robustness of smaller semantic parsers can be enhanced through adversarial training, this approach is not feasible for large language models in real-world scenarios, as it requires both substantial computational resources and expensive human annotation on in-domain semantic parsing data. This paper presents the first empirical study on the adversarial robustness of a large prompt-based language model of code, \codex. Our results demonstrate that the state-of-the-art (SOTA) code-language models are vulnerable to carefully crafted adversarial examples. To address this challenge, we propose methods for improving robustness without the need for significant amounts of labeled data or heavy computational resources.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑