arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.08669cs.CL
收藏

评估文本数据质量对特征表示和机器学习模型的影响

Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models

School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算技术学院)

Tabinda Sarwar, Antonio Jose Jimeno Yepes, Lawrence Cavedon

AI总结:

本研究开发 token 级文本数据集质量错误率指标,用 Mixtral LLM 量化纠正错误,分析医疗数据集后发现模型对<10%错误率耐受、≥10%性能显著下降,强调数据集质量评估及错误纠正的重要性。

摘要:

背景:受控环境下收集的数据通常会产生高质量数据集,但在实际应用中,数据收集质量常受影响,且数据集质量显著影响机器学习模型性能已得到充分证实。方法:开发了一种基础错误率指标,用于在 token 级别评估文本数据集质量;使用 Mixtral 大型语言模型(LLM)量化并纠正低质量数据集中的错误;研究分析了两个医疗数据集:高质量的 MIMIC-III 公共医院数据集和澳大利亚养老院的较低质量私有数据集;在 MIMIC 中系统性地引入不同比率的错误,同时使用 LLM 提升 ACH 数据集的质量。结果:从 MIMIC 和 ACH 数据集分别抽取 35774 名和 6336 名患者数据,使用 Mixtral 在 MIMIC 中引入错误并在 ACH 中纠正错误;Mixtral 正确检测出 63%的病程记录中的错误,其中 17%因医学术语导致单个 token 分类错误;LLM 展现出通过解决各类错误提升病程记录质量的潜力;在不同错误率下,特征表示性能对较低错误率(<10%)具有耐受性,但在较高错误率时显著下降。结论:研究表明,模型在错误率较低(<10%)的数据集上表现相对良好,但随着错误率升高(≥10%),性能显著下降;因此,在将数据集用于机器学习任务前评估其质量至关重要,对于错误率较高的数据集,实施纠正措施对确保机器学习模型的可靠性和有效性必不可少。

英文摘要:

Background: Data collected in controlled settings typically results in high-quality datasets. However, in real-world applications, the quality of data collection is often compromised. It is well established that the quality of a dataset significantly impacts the performance of machine learning models. Methods: A rudimentary error rate metric was developed to evaluate textual dataset quality at the token level. Mixtral Large Language Model (LLM) was used to quantify and correct errors in low quality datasets. The study analyzed two healthcare datasets: the high-quality MIMIC-III public hospital dataset and a lower-quality private dataset from Australian aged care homes. Errors were systematically introduced into MIMIC at varying rates, while the ACH dataset quality was improved using the LLM. Results: For the sampled 35,774 and 6,336 patients from the MIMIC and ACH datasets respectively, we used Mixtral to introduce errors in MIMIC and correct errors in ACH. Mixtral correctly detected errors in 63% of progress notes, with 17% containing a single token misclassified due to medical terminology. LLMs demonstrated potential for improving progress note quality by addressing various errors. Under varying error rates, feature representation performance was tolerant to lower error rates (<10%) but declined significantly at higher rates. Conclusions: The study revealed that models performed relatively well on datasets with lower error rates (<10%), but their performance declined significantly as error rates increased (>=10%). Therefore, it is crucial to evaluate the quality of a dataset before utilizing it for machine learning tasks. For datasets with higher error rates, implementing corrective measures is essential to ensure the reliability and effectiveness of machine learning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑