arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.04434cs.CLcs.AIcs.HCcs.LG
收藏

指导基准创建的实时的视觉反馈:人与度量在环工作流程

Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow

Arizona State University(亚利桑那州立大学)

Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral, Chris Bryan

摘要:

最近的研究表明,语言模型利用基准测试中的“工件”来解决任务,而不是真正学习它们,从而导致模型性能膨胀。为了创建更好的基准,我们提出了VAIDA,这是一种新的NLP基准创建范例,专注于指导众包工作者,这是解决基准特性的一个未被充分探索的方面。VAIDA通过提供实时视觉反馈和建议来提高样品质量,从而促进样品校正。我们的方法是域,模型,任务和度量不可知的,并构成了一个范式转变,通过人和度量的循环工作流的强大,有效和动态的基准创建。我们通过专家评审和NASA TLX的用户研究进行评估。我们发现VAIDA减少了众包工作者和分析师的努力、挫折感、精神和时间需求,同时提高了两个用户组的性能,使创建的样本中的工件水平降低了45.8%。作为我们用户研究的一个副产品,我们观察到创建的样本在模型之间是对抗性的,导致性能下降31.3%(BERT),22.5%(RoBERTa),14.98%(GPT-3 fewshot)。

英文摘要:

Recent research has shown that language models exploit `artifacts' in benchmarks to solve tasks, rather than truly learning them, leading to inflated model performance. In pursuit of creating better benchmarks, we propose VAIDA, a novel benchmark creation paradigm for NLP, that focuses on guiding crowdworkers, an under-explored facet of addressing benchmark idiosyncrasies. VAIDA facilitates sample correction by providing realtime visual feedback and recommendations to improve sample quality. Our approach is domain, model, task, and metric agnostic, and constitutes a paradigm shift for robust, validated, and dynamic benchmark creation via human-and-metric-in-the-loop workflows. We evaluate via expert review and a user study with NASA TLX. We find that VAIDA decreases effort, frustration, mental, and temporal demands of crowdworkers and analysts, simultaneously increasing the performance of both user groups with a 45.8% decrease in the level of artifacts in created samples. As a by product of our user study, we observe that created samples are adversarial across models, leading to decreases of 31.3% (BERT), 22.5% (RoBERTa), 14.98% (GPT-3 fewshot) in performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑