Modern semantic parsers suffer from two principal limitations. First, training requires expensive collection of utterance-program pairs. Second, semantic parsers fail to generalize at test time to new compositions/structures that have not been observed during training. Recent research has shown that automatic generation of synthetic utterance-program pairs can alleviate the first problem, but its potential for the second has thus far been under-explored. In this work, we investigate automatic generation of synthetic utterance-program pairs for improving compositional generalization in semantic parsing. Given a small training set of annotated examples and an "infinite" pool of synthetic examples, we select a subset of synthetic examples that are structurally-diverse and use them to improve compositional generalization. We evaluate our approach on a new split of the schema2QA dataset, and show that it leads to dramatic improvements in compositional generalization as well as moderate improvements in the traditional i.i.d setup. Moreover, structurally-diverse sampling achieves these improvements with as few as 5K examples, compared to 1M examples when sampling uniformly at random -- a 200x improvement in data efficiency.
Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
New York University（纽约大学）；ADAPT Centre, DCU（都柏林城市大学ADAPT中心）；IBM Research（IBM研究院）；Google（谷歌）；Google Research（谷歌研究院）；Yale University（耶鲁大学）；trivago N.V.（途易集团）；University of Edinburgh（爱丁堡大学）；Allen Institute for AI（艾伦人工智能研究所）
Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
Center for Alzheimer’s Disease and Related Dementias, NIA, NIH（阿尔茨海默病及相关痴呆症研究中心，国家老龄化研究所，国家卫生研究院）；DataTecnica LLC（DataTecnica公司）；Johns Hopkins University（约翰霍普金斯大学）；Laboratory of Neurogenetics, NIA, NIH（神经遗传学实验室，国家老龄化研究所，国家卫生研究院）
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；College of Design and Innovation, Tongji University（同济大学设计与创新学院）
Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）；LUMLIS (Lisbon ELLIS Unit)（LUMLIS（里斯本ELLIS单元））；University of Edinburgh（爱丁堡大学）；Unbabel（Unbabel公司）
A Controllable Examination for Long-Context Language Models
University of Edinburgh（爱丁堡大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；Alibaba Group（阿里巴巴集团）；University of Amsterdam（阿姆斯特丹大学）
NeMo: Needle in a Montage for Video-Language Understanding
The Chinese University of Hong Kong（香港中文大学）；Phoenix TV（凤凰电视台）；Stanford University（斯坦福大学）；University of Liverpool（利物浦大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training
Swamynathan V P Department of Computer Science and Engineering Amrita School of Computing Amrita Vishwa Vidyapeetham Chennai, Tamil Nadu, India（斯瓦米纳坦·V·P 计算机科学与工程系 阿米塔学校 阿米塔大学 芬治, 印度泰米尔纳德邦）
