arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.19133cs.CL
收藏

评估基于网络摄像头的注视数据作为人工原理标注的替代方案

Evaluating Webcam-based Gaze Data as an Alternative for Human Rationale Annotations

University of Copenhagen(哥本哈根大学);Technische Universität Berlin(柏林工业大学);IT University of Copenhagen(哥本哈根信息技术大学);University of Zurich(苏黎世大学)

Stephanie Brandl, Oliver Eberle, Tiago Ribeiro, Anders Søgaard, Nora Hollenstein

AI总结:

本文研究网络摄像头眼动数据能否替代人工原理标注,通过WebQAmGaze数据集比较多语言Transformer模型的重要性分数,发现注视数据可揭示语言信息并给出相近的方法排序。

摘要:

手动标注的输入片段形式的原理通常在评估NLP中的可解释性方法时作为基准真值。然而,它们耗时且常受标注过程偏差影响。本文探讨以网络摄像头眼动记录形式呈现的人类注视,在评估重要性分数时是否构成有效替代。我们评估注视数据提供的额外信息,包括总阅读时间、注视熵,以及相对于人工原理标注的解码准确率。我们将面向信息检索问答的多语言数据集WebQAmGaze,与4种不同的多语言基于Transformer的语言模型(mBERT、distil-mBERT、XLMR和XLMR-L)及3种语言(英语、西班牙语和德语)的注意力和基于可解释性的重要性分数进行比较。我们的流程可轻松应用于其他任务和语言。研究结果表明,注视数据提供了宝贵的语言学洞察,可用于推断任务难度,并进一步显示出与人工原理相当的可解释性方法排序。

英文摘要:

Rationales in the form of manually annotated input spans usually serve as ground truth when evaluating explainability methods in NLP. They are, however, time-consuming and often biased by the annotation process. In this paper, we debate whether human gaze, in the form of webcam-based eye-tracking recordings, poses a valid alternative when evaluating importance scores. We evaluate the additional information provided by gaze data, such as total reading times, gaze entropy, and decoding accuracy with respect to human rationale annotations. We compare WebQAmGaze, a multilingual dataset for information-seeking QA, with attention and explainability-based importance scores for 4 different multilingual Transformer-based language models (mBERT, distil-mBERT, XLMR, and XLMR-L) and 3 languages (English, Spanish, and German). Our pipeline can easily be applied to other tasks and languages. Our findings suggest that gaze data offers valuable linguistic insights that could be leveraged to infer task difficulty and further show a comparable ranking of explainability methods to that of human rationales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑