Collecting labeled datasets in finance is challenging due to scarcity of domain experts and higher cost of employing them. While Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, their effectiveness on domain specific datasets remains underexplored. To address this gap, we investigate the potential of LLMs as efficient data annotators for extracting relations in financial documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2, and MPT Instruct) against expert annotators and crowdworkers. We demonstrate that the current state-of-the-art LLMs can be sufficient alternatives to non-expert crowdworkers. We analyze models using various prompts and parameter settings and find that customizing the prompts for each relation group by providing specific examples belonging to those groups is paramount. Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify outputs that may require expert attention. Finally, we perform an extensive time, cost and error analysis and provide recommendations for the collection and usage of automated annotations in domain-specific settings.
FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
State Key Laboratory for Multimedia Information Processing, PKU-Anker LLM Lab（多媒体信息处理国家重点实验室，PKU-Anker LLM实验室）；School of Computer Science, Peking University（北京大学计算机学院）；HKUST（香港科技大学）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Washington（华盛顿大学）
Dalian University of Technology（大连理工大学）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；Tencent（腾讯）；Tongji University（同济大学）
Knowledge-Augmented Large Language Model Agents for Explainable Financial Decision-Making
Boston University（波士顿大学）；Hofstra University（霍夫斯特拉大学）；University of Southern California（南加州大学）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）
LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics
Visual Analysis and Perception Lab（视觉分析与感知实验室）；Pioneer Center for AI（先锋人工智能中心）；Center of Excellence for Global Mobility Law（全球移动法律卓越中心）；Department of Computer Science（计算机科学系）
Faculty of Information Science；Technology Hokkaido University；School of Engineering The University of Tokyo 0000-0001-8519-5617；Technology Hokkaido University 0000-0001-5030-625X；Technology Hokkaido University 0000-0003-1987-5336
Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents
Corpus Considerations for Annotator Modeling and Scaling
University of Marburg（马尔堡大学）；Bonn-Aachen International Center for Information Technology (b-it), University of Bonn（波恩-亚琛信息技术国际中心（b-it），波恩大学）；University of Mannheim（曼海姆大学）
AI Adoption to Combat Financial Crime: Study on Natural Language Processing in Adverse Media Screening of Financial Services in English and Bangla multilingual interpretation
Institute of Business Administration - JU（贾达沃普尔大学商业管理学院）
Application of Natural Language Processing in Financial Risk Detection
Washington University in St. Louis（圣路易斯华盛顿大学）；Columbia University（哥伦比亚大学）；University of Electronic Science and Technology of China（电子科技大学）；The University of Chicago（芝加哥大学）；University of Houston（休斯顿大学）
FinS-Pilot: A Benchmark for Online Financial RAG System
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高陵人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程研究中心，教育部）；Tencent Inc.（腾讯公司）
CreditARF: A Framework for Corporate Credit Rating with Annual Report and Financial Feature Integration
School of Cyberspace Security, Beijing University of Posts and Telecommunications（网络安全学院，北京邮电大学）；Guotai Junan Securities（国泰君安证券）；Hebei University of Technology（河北工业大学）
