作为科学工具的自动分类器:离真值更进一步
Automatic Classifiers as Scientific Instruments: One Step Further Away from Ground-Truth
摘要:
基于自动机器学习的各种心理和社会现象的检测器(例如,情绪、压力、参与)对推进基础科学有很大的潜力。然而,当检测器$d$被训练为近似现有的测量工具(例如,调查表、观察协议),则在解释使用$d$收集的测量值时必须小心,因为它们与基础结构进一步分离。我们检查$d$的准确性如何影响$U$之间的估计相关性(例如,应力)和一些其它现象$V$(例如,学术成绩)。特别是:(1)我们证明,如果$U$和$V$之间的真实相关性是$r$,则在与$U$的相关性是$q$的所有向量$\mathcal{T}^n$上的期望样本相关性是$qr$。(2)我们推导出一个公式的概率,样本相关性(超过$n$科目)使用$d$是积极的,因为真正的相关性是负的(反之亦然);这种概率可以是实质性的(周围$20-30\%$)的值$n$和$q$已被用于最近的情感计算研究。%我们还表明,这种概率在$n$和$q$中单调下降。(3)为了减少由自动检测器估计的相关性的方差,我们证明了使用不同的训练架构和超参数训练多个神经网络$d^{(1)},\ldots,d^{(m)}$,用于相同的检测任务,只提供有限的“覆盖范围”的$\mathcal{T}^n$。
英文摘要:
Automatic machine learning-based detectors of various psychological and social phenomena (e.g., emotion, stress, engagement) have great potential to advance basic science. However, when a detector $d$ is trained to approximate an existing measurement tool (e.g., a questionnaire, observation protocol), then care must be taken when interpreting measurements collected using $d$ since they are one step further removed from the underlying construct. We examine how the accuracy of $d$, as quantified by the correlation $q$ of $d$'s outputs with the ground-truth construct $U$, impacts the estimated correlation between $U$ (e.g., stress) and some other phenomenon $V$ (e.g., academic performance). In particular: (1) We show that if the true correlation between $U$ and $V$ is $r$, then the expected sample correlation, over all vectors $\mathcal{T}^n$ whose correlation with $U$ is $q$, is $qr$. (2) We derive a formula for the probability that the sample correlation (over $n$ subjects) using $d$ is positive given that the true correlation is negative (and vice-versa); this probability can be substantial (around $20-30\%$) for values of $n$ and $q$ that have been used in recent affective computing studies. %We also show that this probability decreases monotonically in $n$ and in $q$. (3) With the goal to reduce the variance of correlations estimated by an automatic detector, we show that training multiple neural networks $d^{(1)},\ldots,d^{(m)}$ using different training architectures and hyperparameters for the same detection task provides only limited ``coverage'' of $\mathcal{T}^n$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。