The rapid rollout of AI in heterogeneous public and societal sectors has subsequently escalated the need for compliance with regulatory standards and frameworks. The EU AI Act has emerged as a landmark in the regulatory landscape. The development of solutions that elicit the level of AI systems' compliance with such standards is often limited by the lack of resources, hindering the semi-automated or automated evaluation of their performance. This generates the need for manual work, which is often error-prone, resource-limited or limited to cases not clearly described by the regulation. This paper presents an open, transparent, and reproducible method of creating a resource that facilitates the evaluation of NLP models with a strong focus on RAG systems. We have developed a dataset that contain the tasks of risk-level classification, article retrieval, obligation generation, and question-answering for the EU AI Act. The dataset files are in a machine-to-machine appropriate format. To generate the files, we utilise domain knowledge as an exegetical basis, combining with the processing and reasoning power of large language models to generate scenarios along with the respective tasks. Our methodology demonstrates a way to harness language models for grounded generation with high document relevancy. Besides, we overcome limitations such as navigating the decision boundaries of risk-levels that are not explicitly defined within the EU AI Act, such as limited and minimal cases. Finally, we demonstrate our dataset's effectiveness by evaluating a RAG-based solution that reaches 0.87 and 0.85 F1-score for prohibited and high-risk scenarios.
Retrieval-Augmented Generation for Large Language Models: A Survey
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）；College of Design and Innovation, Tongji University（同济大学设计与创新学院）
RAG-BioQA: A Retrieval-Augmented Generation Framework for Long-Form Biomedical Question Answering
Department of CSE Emory University Georgia, USA（计算机科学与工程系 埃默里大学 美国佐治亚州）；Department of CSE SRM University AP Andhra Pradesh, India（计算机科学与工程系 SRM大学AP 安得拉邦印度）；Department of CSE Trine University Michigan, USA（计算机科学与工程系 特林大学 密歇根州美国）
CheckSupport: A Local LLM-Powered Tool for Automated Manuscript Submission Checklist Selection and Completion
Department of Radiology, Perelman School of Medicine at University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学学院放射科）；Department of Computer Science, Drexel University（德雷塞尔大学计算机科学系）；Department of Computer and Information Science, School of Engineering and Applied Science, University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院计算机与信息科学系）；Department of Radiology, Cooper University Hospital（科珀大学医院放射科）；University of California Davis Graduate School of Management（加州大学戴维斯分校管理研究生院）
NyayaMind- A Framework for Transparent Legal Reasoning and Judgment Prediction in the Indian Legal System
IIT Kanpur, India（印度理工学院坎普尔分校）；IIT Kharagpur, India（印度理工学院克勒格布尔分校）；Symbiosis Law School Pune, India（印度共生法学院浦那分校）；Dattam Labs, India（印度Dattam实验室）；University of Birmingham, Dubai, United Arab Emirates（英国伯明翰大学迪拜校区）
A Lightweight LLM Framework for Disaster Humanitarian Information Classification
Department of Civil, Architectural & Environment Engineering, Sungkyunkwan University（苏州市立大学土木、建筑与环境工程系）；School of Geography, University of Leeds（利兹大学地理学院）
Reproducible Benchmarking for Lung Nodule Detection and Malignancy Classification Across Multiple Low-Dose CT Datasets
Duke University School of Medicine（杜克大学医学院）；Pratt School of Engineering, Duke University（杜克大学普拉特工程学院）；University of Colorado（科罗拉多大学）；Duke University（杜克大学）
