arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.08115cs.LGstat.AP
收藏

机器学习模型错误遏制关注规则的自动生成

Automatic Generation of Attention Rules For Containment of Machine Learning Model Errors

Samuel Ackerman, Axel Bendavid, Eitan Farchi, Orna Raz

AI总结:

本文提出自动生成关注规则以分离机器学习模型最可能出错样本,采用基于特征切片的策略,并引入多指标评估与TOPSIS聚合,优于常见基线。

摘要:

机器学习(ML)解决方案在许多应用中普遍存在。然而,在使这些解决方案达到业务级标准方面存在诸多挑战。例如,将底层ML模型的错误率维持在可接受的低水平。通常,特征输入与待预测目标特征之间的真实关系是不确定的,因此具有统计性质。我们提出的方法是将最可能被错误预测的观测值分离到“关注集”中。这些关注集可直接有助于模型诊断和改进,并可用于为这些有问题的观测值决定替代行动方案。我们提出了几种算法(“策略”)来确定分离这些观测值的最优规则。特别是,我们倾向于使用基于特征切片的策略,因为它们具有人类可解释性、模型无关性,并且需要最少的补充输入或知识。此外,我们表明这些策略优于几种常见基线,例如选择预测置信度低于阈值的观测值。为了评估策略,我们引入了衡量各种期望质量的指标,例如其性能、稳定性以及对未见数据的泛化能力;这些策略在几个公开可用的数据集上进行了评估。我们使用TOPSIS(一种多准则决策方法)将这些指标聚合为每个策略的单一质量分数,以便进行比较。

英文摘要:

Machine learning (ML) solutions are prevalent in many applications. However, many challenges exist in making these solutions business-grade. For instance, maintaining the error rate of the underlying ML models at an acceptably low level. Typically, the true relationship between feature inputs and the target feature to be predicted is uncertain, and hence statistical in nature. The approach we propose is to separate the observations that are the most likely to be predicted incorrectly into 'attention sets'. These can directly aid model diagnosis and improvement, and be used to decide on alternative courses of action for these problematic observations. We present several algorithms (`strategies') for determining optimal rules to separate these observations. In particular, we prefer strategies that use feature-based slicing because they are human-interpretable, model-agnostic, and require minimal supplementary inputs or knowledge. In addition, we show that these strategies outperform several common baselines, such as selecting observations with prediction confidence below a threshold. To evaluate strategies, we introduce metrics to measure various desired qualities, such as their performance, stability, and generalizability to unseen data; the strategies are evaluated on several publicly-available datasets. We use TOPSIS, a Multiple Criteria Decision Making method, to aggregate these metrics into a single quality score for each strategy, to allow comparison.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑