Automated text annotation is a compelling use case for generative large language models (LLMs) in social media research. Recent work suggests that LLMs can achieve strong performance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered framework for responsibly evaluating artificial intelligence tools used in automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets from recently published computational social science articles in high-impact journals. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Although the quality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even within datasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimization strategies such as prompt tuning. Grounding automated annotation in validation labels generated by humans is essential for responsible evaluation.
AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions
Media Informatics Group, University of Regensburg, Regensburg, Germany(里根斯堡大学媒体信息学组);Information Science Group, University of Regensburg, Regensburg, Germany(里根斯堡大学信息科学组)
Annotator-Centric Active Learning for Subjective NLP Tasks
Idiap Research Institute(伊迪亚普研究所);Leiden Institute of Advanced Computer Science, Leiden University(莱顿大学莱顿高级计算机科学学院);Institute for Natural Language Processing, University of Stuttgart(斯图加特大学自然语言处理研究所);Interactive Intelligence, TU Delft(代尔夫特理工大学交互智能研究所)
No Language Left Behind: Scaling Human-Centered Machine Translation
Meta AI;UC Berkeley(加州大学伯克利分校);Johns Hopkins University(约翰斯·霍普金斯大学)
在全球范围内消除语言障碍的目标的推动下,机器翻译已经巩固了自己作为当今人工智能研究的一个关键焦点。然而,这些努力都集中在一小部分语言上,留下了绝大多数低资源语言。如何才能在确保安全、高质量的结果的同时,打破200种语言的障碍,同时考虑到道德因素?在《No Language Left Behind》中,我们首先通过与母语人士的探索性访谈,将对低资源语言翻译支持的需求置于情境中,从而接受了这一挑战。然后,我们创建了数据集和模型,旨在缩小低资源语言和高资源语言之间的性能差距。更具体地说,我们开发了一个条件计算模型的基础上稀疏门控的混合专家模型,是训练的数据与新的和有效的数据采掘技术,为低资源的语言。我们提出了多个架构和训练改进,以抵消过度拟合,同时训练数千个任务。重要的是,我们使用人类翻译的基准Flores-200评估了超过40,000种不同翻译方向的性能,并将人类评估与Flores-200中涵盖所有语言的新型毒性基准相结合,以评估翻译安全性。我们的模型实现了44%的BLEU相对于以前的国家的最先进的改进,奠定了重要的基础,实现一个通用的翻译系统。最后,我们开源了本工作中描述的所有贡献,可在https://github.com/facebookresearch/fairseq/tree/nllb上访问。
IBM Almaden Research Center(IBM阿尔马登研究中心);Rensselaer Polytechnic Institute(伦斯勒理工学院);Virginia Polytechnic Institute and State University(弗吉尼亚理工学院暨州立大学);MIT Critical Data(麻省理工学院关键数据实验室);Albert Einstein Healthcare Network-Philadelphia Campus(阿尔伯特·爱因斯坦医疗网络费城院区);Harvard Medical School(哈佛医学院)