Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
New York University(纽约大学);ADAPT Centre, DCU(都柏林城市大学ADAPT中心);IBM Research(IBM研究院);Google(谷歌);Google Research(谷歌研究院);Yale University(耶鲁大学);trivago N.V.(途易集团);University of Edinburgh(爱丁堡大学);Allen Institute for AI(艾伦人工智能研究所)
Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad Mahamood, Sebastian Gehrmann, Miruna Clinciu, Khyathi Chandu, João Sedoc
To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.
Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
Center for Alzheimer’s Disease and Related Dementias, NIA, NIH(阿尔茨海默病及相关痴呆症研究中心,国家老龄化研究所,国家卫生研究院);DataTecnica LLC(DataTecnica公司);Johns Hopkins University(约翰霍普金斯大学);Laboratory of Neurogenetics, NIA, NIH(神经遗传学实验室,国家老龄化研究所,国家卫生研究院)
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);College of Design and Innovation, Tongji University(同济大学设计与创新学院)
Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation
Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico(高等技术学院);LUMLIS (Lisbon ELLIS Unit)(LUMLIS(里斯本ELLIS单元));University of Edinburgh(爱丁堡大学);Unbabel(Unbabel公司)