arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.14255cs.AIcs.MA
收藏

通过信任分数和数据增强提高ML模型可信度的人在回路在线多智能体方法

Human-in-the-loop online multi-agent approach to increase trustworthiness in ML models through trust scores and data augmentation

Barcelona Supercomputing Center(巴塞罗那超级计算中心);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学);Lenovo Infrastructure Solutions Group, Lenovo(联想基础设施解决方案集团(联想))

Gusseppe Bravo-Rocca, Peini Liu, Jordi Guitart, Ajay Dholakia, David Ellison, Miroslav Hodak

摘要:

提高ML模型的准确性是不够的,我们还必须提高其可信度。这是为汽车、金融和医疗保健等安全关键型应用构建弹性AI系统的重要一步。为此,我们提出了一个多智能体系统,结合了机器和人类代理。在该系统中,检查器代理使用基于一致性的方法计算每个实例的信任分数(其惩罚预测中的过度自信和过度谨慎)并对其进行排名;然后改进器代理基于基于人类规则的过程(其被认为是安全的)过滤异常实例,获得人类标签,应用几何数据增强,并使用迁移学习用增强的数据进行再训练。我们评估系统的损坏版本的MNIST和FashionMNIST数据集。与基线方法相比,我们只需增加一些标签就可以提高准确性和信任度。

英文摘要:

Increasing a ML model accuracy is not enough, we must also increase its trustworthiness. This is an important step for building resilient AI systems for safety-critical applications such as automotive, finance, and healthcare. For that purpose, we propose a multi-agent system that combines both machine and human agents. In this system, a checker agent calculates a trust score of each instance (which penalizes overconfidence and overcautiousness in predictions) using an agreement-based method and ranks it; then an improver agent filters the anomalous instances based on a human rule-based procedure (which is considered safe), gets the human labels, applies geometric data augmentation, and retrains with the augmented data using transfer learning. We evaluate the system on corrupted versions of the MNIST and FashionMNIST datasets. We get an improvement in accuracy and trust score with just few additional labels compared to a baseline approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑