This work presents a custom approach to developing a domain specific knowledge assistant for sustainability reporting using the International Financial Reporting Standards (IFRS). In this domain, there is no publicly available question-answer dataset, which has impeded the development of a high-quality chatbot to support companies with IFRS reporting. The two key contributions of this project therefore are:
(1) A high-quality synthetic question-answer (QA) dataset based on IFRS sustainability standards, created using a novel generation and evaluation pipeline leveraging Large Language Models (LLMs). This comprises 1,063 diverse QA pairs that address a wide spectrum of potential user queries in sustainability reporting. Various LLM-based techniques are employed to create the dataset, including chain-of-thought reasoning and few-shot prompting. A custom evaluation framework is developed to assess question and answer quality across multiple dimensions, including faithfulness, relevance, and domain specificity. The dataset averages a score range of 8.16 out of 10 on these metrics.
(2) Two architectures for question-answering in the sustainability reporting domain - a RAG pipeline and a fully LLM-based pipeline. The architectures are developed by experimenting, fine-tuning, and training on the QA dataset. The final pipelines feature an LLM fine-tuned on domain specific data and an industry classification component to improve the handling of complex queries. The RAG architecture achieves an accuracy of 85.32% on single-industry and 72.15% on cross-industry multiple-choice questions, outperforming the baseline approach by 4.67 and 19.21 percentage points, respectively. The LLM-based pipeline achieves an accuracy of 93.45% on single-industry and 80.30% on cross-industry multiple-choice questions, an improvement of 12.80 and 27.36 percentage points over the baseline, respectively.
Domain-Specific Improvement on Psychotherapy Chatbot Using Assistant
Czech Technical University in Prague（布拉格捷克技术大学）；National Institute of Mental Health in Prague（布拉格国家精神卫生研究所）；The University of Hong Kong（香港大学）；Shenzhen Mental Health Centre（深圳市精神卫生中心）
Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能跨学科计划，首尔国立大学）；AIIS, ASRI, INMC, and ISRC, Seoul National University（人工智能研究所、人工智能研究机构、智能网络中心和智能系统研究组，首尔国立大学）
Cog-TiPRO: Iterative Prompt Refinement with LLMs to Detect Cognitive Decline via Longitudinal Voice Assistant Commands
Computer Science, University of Massachusetts, Boston, MA, USA（计算机科学，马萨诸塞大学，波士顿，马萨诸塞州，美国）；School of Medicine, University of North Carolina, Chapel Hill, NC, USA（医学院，北卡罗来纳大学，夏洛特，北卡罗来纳州，美国）
T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task
Artificial Intelligence Innovation Center, Yangtze Delta Region Institute of Tsinghua University（清华大学长三角研究院人工智能创新中心）；Nanyang Technological University（南洋理工大学）
Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators
Department of Computer Science, University of Maryland（大学计算机科学系）；National Consortium for the Study of Terrorism and Responses to Terrorism（反恐与反恐响应国家研究中心）
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电子工程与计算机科学系）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
An AI-Powered Research Assistant in the Lab: A Practical Guide for Text Analysis Through Iterative Collaboration with LLMs
Universidad Externado de Colombia（哥伦比亚外部大学）；Universidad de los Andes（安第斯大学）；University of Michigan（密歇根大学）；Rice University（莱斯大学）；University of Toronto（多伦多大学）；University of Southampton（南安普顿大学）
Bridge-AI Lab（Bridge-AI实验室）；University of Central Florida（中佛罗里达大学）；Auburn University（奥本大学）；Wichita State University（威奇托州立大学）；Gustavus Adolphus College（古斯塔夫斯阿道夫学院）
Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
Peking University（北京大学）；Peking University Cancer Hospital and Institute（北京大学肿瘤医院暨北京市肿瘤防治研究所）；National Engineering Laboratory for Big Data Analysis and Applications, Peking University（北京大学大数据分析与应用技术国家工程实验室）；Peking University Changsha Institute for Computing and Digital Economy（北京大学长沙计算与数字经济研究院）
