arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.00201cs.LGcs.AI
收藏

无监督机器学习模型选择中的主观性

Subjectivity in Unsupervised Machine Learning Model Selection

Duke University(杜克大学);George Mason University(乔治梅森大学)

Wanyi Chen, Mary L. Cummings

AI总结:

本研究以Hidden Markov Model为例,通过33名参与者和三个LLMs在三种场景下的模型选择实验,揭示了无监督学习中模型选择的主观性、变异性和不一致性,强调需建立标准化记录方法。

摘要:

模型选择是无监督机器学习的必要步骤。尽管有大量标准和指标,模型选择仍具主观性。高度主观性可能引发对各类机器学习研究可重复性和再现性的质疑,并对现实世界中部署模型的稳健性产生怀疑。然而,建模者偏好对模型选择结果的影响在很大程度上仍未被探索。本研究以Hidden Markov Model为例,调查模型选择中涉及的主观性。我们要求33名参与者和三个大型语言模型(LLMs)在三种场景下进行模型选择。结果显示,参与者和LLMs的选择均存在变异性和不一致性,尤其在不同标准和指标不一致时。主观性来源包括对不同标准和指标重要性的不同看法、对模型应多简约的不同观点,以及数据集大小应如何影响模型选择的不同看法。结果强调了开发更标准化方法来记录模型选择过程中主观选择的重要性。

英文摘要:

Model selection is a necessary step in unsupervised machine learning. Despite numerous criteria and metrics, model selection remains subjective. A high degree of subjectivity may lead to questions about repeatability and reproducibility of various machine learning studies and doubts about the robustness of models deployed in the real world. Yet, the impact of modelers' preferences on model selection outcomes remains largely unexplored. This study uses the Hidden Markov Model as an example to investigate the subjectivity involved in model selection. We asked 33 participants and three Large Language Models (LLMs) to make model selections in three scenarios. Results revealed variability and inconsistencies in both the participants' and the LLMs' choices, especially when different criteria and metrics disagree. Sources of subjectivity include varying opinions on the importance of different criteria and metrics, differing views on how parsimonious a model should be, and how the size of a dataset should influence model selection. The results underscore the importance of developing a more standardized way to document subjective choices made in model selection processes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑