While humans can extract information from unstructured text with high precision and recall, this is often too time-consuming to be practical. Automated approaches, on the other hand, produce nearly-immediate results, but may not be reliable enough for high-stakes applications where precision is essential. In this work, we consider the benefits and drawbacks of various human-only, human-machine, and machine-only information extraction approaches. We argue for the utility of a human-in-the-loop approach in applications where high precision is required, but purely manual extraction is infeasible. We present a framework and an accompanying tool for information extraction using weak-supervision labelling with human validation. We demonstrate our approach on three criminal justice datasets. We find that the combination of computer speed and human understanding yields precision comparable to manual annotation while requiring only a fraction of time, and significantly outperforms fully automated baselines in terms of precision.
InteractiveIE: Towards Assessing the Strength of Human-AI Collaboration in Improving the Performance of Information Extraction
University of Maryland（马里兰大学）；Amazon（亚马逊公司）；Adobe Research（奥多比研究院）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；John Hopkins University（约翰霍普金斯大学）；University of Maryland School of Law（马里兰大学法学院）
Multimodal intelligibility of scholarly hypertext: the documentalist's contribution. A required collaboration for serial documentisation in the scientific editorial process
