arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1504.06329cs.LGcs.CLstat.ML
收藏

基于稳定预测的停止主动学习分析

Analysis of Stopping Active Learning based on Stabilizing Predictions

Center for Advanced Study of Language(语言高级研究中心);University of Maryland(马里兰大学);Raytheon BBN Technologies(雷神BBN技术公司)

Michael Bloodgood, John Grothendieck

摘要:

在自然语言处理(NLP)社区中,主动学习已经被广泛研究和应用,以减轻新的NLP系统和技术的开发者所面临的注释瓶颈。本文首次提出了基于稳定化预测(SP)的停止主动学习的理论分析。分析揭示了SP方法成功的三个核心要素:(1)连续训练模型之间Cohen Kappa一致性的界限对模型的F度量性能差异施加了界限;(2)由于止动装置不必被标记,因此在实践中可以将其制造得很大,有助于保证结果在测试/应用时转移到先前看不见的示例流;以及(3)可以获得连续模型之间的Kappa的良好(低方差)样本估计。的Kappa协议和连续模型之间的性能差异的水平之间的关系的证明。具体地,如果两个模型之间的Kappa一致性超过阈值T(其中$T>0$),则在所有情况下,这些模型之间的F测量性能的差异都以$\frac{4(1-T)}{T}$为界。如果模型的正合取精度被假定为$p$,则边界可以被收紧为$\frac{4(1-T)}{(p+1)T}$。

英文摘要:

Within the natural language processing (NLP) community, active learning has been widely investigated and applied in order to alleviate the annotation bottleneck faced by developers of new NLP systems and technologies. This paper presents the first theoretical analysis of stopping active learning based on stabilizing predictions (SP). The analysis has revealed three elements that are central to the success of the SP method: (1) bounds on Cohen's Kappa agreement between successively trained models impose bounds on differences in F-measure performance of the models; (2) since the stop set does not have to be labeled, it can be made large in practice, helping to guarantee that the results transfer to previously unseen streams of examples at test/application time; and (3) good (low variance) sample estimates of Kappa between successive models can be obtained. Proofs of relationships between the level of Kappa agreement and the difference in performance between consecutive models are presented. Specifically, if the Kappa agreement between two models exceeds a threshold T (where $T>0$), then the difference in F-measure performance between those models is bounded above by $\frac{4(1-T)}{T}$ in all cases. If precision of the positive conjunction of the models is assumed to be $p$, then the bound can be tightened to $\frac{4(1-T)}{(p+1)T}$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑