arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.13140cs.CLcs.AI
收藏

DuTrust:一个用于可信性评估的情感分析数据集

DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation

Baidu Inc.(百度公司)

Lijie Wang, Hao Liu, Shuyuan Peng, Hongxuan Tang, Xinyan Xiao, Ying Chen, Hua Wu, Haifeng Wang

摘要:

虽然深度学习模型大大提高了大多数人工智能任务的性能,但由于黑箱问题,它们经常被批评为不可信。因此,已经提出了许多工作来研究深度学习的可信度。然而,由于大多数开放数据集是为评估模型输出的准确性而设计的,因此仍然缺乏用于评估神经网络内部工作的适当数据集。数据集的缺乏显然阻碍了可信性研究的发展。因此,为了系统地评估构建可信系统的因素,我们提出了一种新的和注释良好的情感分析数据集来评估鲁棒性和可解释性。为了评估这些因素,我们的数据集包含关于实例的挑战性分布,手动对抗实例和情感解释的各种注释。进一步提出了几个评价指标的可解释性和鲁棒性。基于数据集和度量标准,我们对三种典型模型的可信性进行了全面的比较,并研究了准确性,鲁棒性和可解释性之间的关系。我们在\url{https://github/xyz}上发布了这个可信度评估数据集,希望我们的工作能够促进为现实世界的应用程序构建更可信的系统的进展。

英文摘要:

While deep learning models have greatly improved the performance of most artificial intelligence tasks, they are often criticized to be untrustworthy due to the black-box problem. Consequently, many works have been proposed to study the trustworthiness of deep learning. However, as most open datasets are designed for evaluating the accuracy of model outputs, there is still a lack of appropriate datasets for evaluating the inner workings of neural networks. The lack of datasets obviously hinders the development of trustworthiness research. Therefore, in order to systematically evaluate the factors for building trustworthy systems, we propose a novel and well-annotated sentiment analysis dataset to evaluate robustness and interpretability. To evaluate these factors, our dataset contains diverse annotations about the challenging distribution of instances, manual adversarial instances and sentiment explanations. Several evaluation metrics are further proposed for interpretability and robustness. Based on the dataset and metrics, we conduct comprehensive comparisons for the trustworthiness of three typical models, and also study the relations between accuracy, robustness and interpretability. We release this trustworthiness evaluation dataset at \url{https://github/xyz} and hope our work can facilitate the progress on building more trustworthy systems for real-world applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑