ClimRetrieve:面向企业气候披露信息检索的基准数据集
ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures
AI总结:
本文提出ClimRetrieve基准数据集,基于30份可持续发展报告构建超8.5K条气候披露问答对,并研究专家知识与嵌入检索的集成,揭示嵌入在知识密集型气候领域中的关键局限。
摘要:
为了处理企业气候沟通中产生的大量定性数据,利益相关者越来越依赖检索增强生成(RAG)系统。然而,在评估特定领域的信息检索——即答案生成的基础——方面仍存在显著差距。为应对这一挑战,本研究通过考察30份可持续发展报告并设置16个详细的气候相关问题,模拟了可持续发展分析师的典型任务。由此,我们获得了一个包含超过8.5K个唯一问题-来源-答案对的数据集,并按不同相关性级别进行标注。此外,我们利用该数据集开发了一个用例,以研究将专家知识集成到基于嵌入的信息检索中的效果。尽管我们表明融入专家知识是有效的,但我们也概述了嵌入在气候变化沟通等知识密集型下游领域中的关键局限性。
英文摘要:
To handle the vast amounts of qualitative data produced in corporate climate communication, stakeholders increasingly rely on Retrieval Augmented Generation (RAG) systems. However, a significant gap remains in evaluating domain-specific information retrieval - the basis for answer generation. To address this challenge, this work simulates the typical tasks of a sustainability analyst by examining 30 sustainability reports with 16 detailed climate-related questions. As a result, we obtain a dataset with over 8.5K unique question-source-answer pairs labeled by different levels of relevance. Furthermore, we develop a use case with the dataset to investigate the integration of expert knowledge into information retrieval with embeddings. Although we show that incorporating expert knowledge works, we also outline the critical limitations of embeddings in knowledge-intensive downstream domains like climate change communication.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。