The rise of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) has rapidly increased the need for high-quality, curated information retrieval datasets. These datasets, however, are currently created with off-the-shelf annotation tools that make the annotation process complex and inefficient. To streamline this process, we developed a specialized annotation tool - AIANO. By adopting an AI-augmented annotation workflow that tightly integrates human expertise with LLM assistance, AIANO enables annotators to leverage AI suggestions while retaining full control over annotation decisions. In a within-subject user study ($n = 15$), participants created question-answering datasets using both a baseline tool and AIANO. AIANO nearly doubled annotation speed compared to the baseline while being easier to use and improving retrieval accuracy. These results demonstrate that AIANO's AI-augmented approach accelerates and enhances dataset creation for information retrieval tasks, advancing annotation capabilities in retrieval-intensive domains.
Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室，计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Baidu Inc.（百度公司）
DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；Swiss Finance Institute (SFI)（瑞士金融研究所）；University of Electronic Science and Technology of China（电子科技大学）
A Question Answering Dataset for Temporal-Sensitive Retrieval-Augmented Generation
Laboratory for Big Data and Decision, National University of Defense Technology, Changsha, China（大数据与决策实验室，国防科技大学，长沙，中国）；Baidu Inc., Beijing, China（百度公司，北京，中国）；Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China（计算机科学与技术系，哈尔滨工业大学（深圳），深圳，中国）
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
In the last decade, the field of Neural Language Modelling has witnessed enormous changes, with the development of novel models through the use of Transformer architectures. However, even these models struggle to model long sequences due to memory constraints and increasing computational complexity. Coreference annotations over the training data can provide context far beyond the modelling limitations of such language models. In this paper we present an extension over the Transformer-block architecture used in neural language models, specifically in GPT2, in order to incorporate entity annotat…
Data Augmented Pipeline for Legal Information Extraction and Reasoning
Center for Juris-Informatics, ROIS-DS（司法信息中心，ROIS-DS）；Japan Advanced Institute of Science and Technology（日本先进科学研究所）；Research and Development Center for LLMs, NII（大语言模型研发中心，NII）
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Measuring event salience is essential in the understanding of stories. This paper takes a recent unsupervised method for salience detection derived from Barthes Cardinal Functions and theories of surprise and applies it to longer narrative forms. We improve the standard transformer language model by incorporating an external knowledgebase (derived from Retrieval Augmented Generation) and adding a memory mechanism to enhance performance on longer works. We use a novel approach to derive salience annotation using chapter-aligned summaries from the Shmoop corpus for classic literary works. Our ev…
Retrieval-Augmented Detection of Potentially Abusive Clauses in Chilean Terms of Service
School of Computer Engineering, Pontificia Universidad Católica de Valparaíso（Pontificia Universidad Católica de Valparaíso计算机工程学院）；Faculty of Law, Universidad Adolfo Ibáñez（Adolfo Ibáñez大学法学院）
On-the-fly Denoising for Data Augmentation in Natural Language Understanding
Hong Kong University of Science and Technology（香港科技大学）；University of Southern California（南加州大学）；University of Cambridge（剑桥大学）；Tencent AI Lab, Seattle（腾讯AI实验室（西雅图））；University of California, Davis（加州大学戴维斯分校）
Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation
University of Aberdeen（阿伯丁大学）；Nanyang Technological University（南洋理工大学）；University College London（伦敦大学学院）；University of Colorado Anschutz Medical Campus（科罗拉多大学安舒兹医学校区）；University of Sheffield（谢菲尔德大学）
GraphRAG-Causal: A novel graph-augmented framework for causal reasoning and annotation in news
Dept. of Artificial Intelligence & Data Science FAST School of Computing NUCES（人工智能与数据科学系 FAST 计算学院 NUCES）；Dept. of Computing and Electronics Engineering Middle East College（计算与电子工程系 中东学院）；Institute for Analytics and Data Science University of Essex（分析与数据科学研究所 埃塞克斯大学）
Annotation guidelines used to guide the annotation of training and evaluation datasets can have a considerable impact on the quality of machine learning models. In this study, we explore the effects of annotation guidelines on the quality of app feature extraction models. As a main result, we propose several changes to the existing annotation guidelines with a goal of making the extracted app features more useful and informative to the app developers. We test the proposed changes via simulating the application of the new annotation guidelines and then evaluating the performance of the supervis…
Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults
Lawrence Bloomberg Faculty of Nursing, University of Toronto（多伦多大学劳伦斯·布隆伯格护理学院）；KITE Research Institute, Toronto Rehabilitation Institute, University Health Network（大学健康网络多伦多康复研究所KITE研究所）；College of Engineering and Technology, American University of the Middle East（中东美国大学工程与技术学院）
