arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.11368cs.CL
收藏

VECHR:一个用于欧洲人权法院脆弱性类型可解释且鲁棒分类的数据集

VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights

Technical University of Munich(慕尼黑工业大学);LMU Munich(慕尼黑大学);Graduate Institute of International and Development Studies(日内瓦高级国际关系及发展学院);University of Zürich(苏黎世大学)

Shanshan Xu, Leon Staufer, T. Y. S. S Santosh, Oana Ichim, Corina Heri, Matthias Grabmair

AI总结:

本文针对欧洲人权法院脆弱性概念难以捉摸且缺乏NLP研究的问题,提出了专家标注的多标签数据集VECHR,涵盖脆弱性类型分类与解释依据,并基准测试了最先进模型,揭示了任务在预测性能、可解释性和OOD鲁棒性上的挑战与改进空间。

摘要:

识别脆弱性对于理解并实施有针对性的支持以赋能有需要的个人至关重要。这在欧洲人权法院(ECtHR)尤为重要,法院在此调整《公约》标准以满足实际的个人需求,从而确保有效的人权保护。然而,脆弱性的概念在ECtHR仍然难以捉摸,且此前没有NLP研究涉及这一问题。为促进该领域的未来研究,我们提出了VECHR,这是一个由专家标注的新型多标签数据集,包含脆弱性类型分类和解释依据。我们从预测和可解释性两个角度在VECHR上对最先进模型的性能进行了基准测试。结果表明了该任务的挑战性,预测性能较低,且模型与专家之间的一致性有限。此外,我们分析了这些模型在处理分布外(OOD)数据时的鲁棒性,观察到整体性能有限。我们的数据集带来了独特的挑战,在性能、可解释性和鲁棒性方面提供了巨大的改进空间。

英文摘要:

Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need. This is especially important at the European Court of Human Rights (ECtHR), where the court adapts Convention standards to meet actual individual needs and thus ensures effective human rights protection. However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it. To enable future research in this area, we present VECHR, a novel expert-annotated multi-label dataset comprising of vulnerability type classification and explanation rationale. We benchmark the performance of state-of-the-art models on VECHR from both prediction and explainability perspectives. Our results demonstrate the challenging nature of the task with lower prediction performance and limited agreement between models and experts. Further, we analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe overall limited performance. Our dataset poses unique challenges offering significant room for improvement regarding performance, explainability, and robustness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑