近来,像 ChatGPT 这样的大语言模型(LLMs)在许多自然语言处理任务上展现出令人印象深刻的零样本性能。本文考察零样本 LLM 在金融领域的有效性。我们将 ChatGPT 及一些开源生成式 LLM 的零样本表现,与在标注数据上微调的 RoBERTa 进行比较。我们围绕数据标注、性能差距以及在金融领域采用生成式模型的可行性,处理三个相互关联的研究问题。我们的发现表明,即使没有标注数据,ChatGPT 也表现良好,但微调模型通常仍优于它。我们的研究还强调,使用生成式模型进行标注可能非常耗时。我们的代码库已在 GitHub 上以 CC BY-NC 4.0 许可公开。
英文摘要:
Recently large language models (LLMs) like ChatGPT have shown impressive performance on many natural language processing tasks with zero-shot. In this paper, we investigate the effectiveness of zero-shot LLMs in the financial domain. We compare the performance of ChatGPT along with some open-source generative LLMs in zero-shot mode with RoBERTa fine-tuned on annotated data. We address three inter-related research questions on data annotation, performance gaps, and the feasibility of employing generative models in the finance domain. Our findings demonstrate that ChatGPT performs well even without labeled data but fine-tuned models generally outperform it. Our research also highlights how annotating with generative models can be time-intensive. Our codebase is publicly available on GitHub under CC BY-NC 4.0 license.
Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contexts
Faculty of Engineering, Thammasat School of Engineering, Thammasat University(工程学院,泰国朱拉安高校工程学院,泰国朱拉安大学);Department of Computer Engineering and Digital Technology, Faculty of Engineering, Chulalongkorn University(计算机工程与数字技术系,工程学院,朱拉安大学);Artificial Intelligence Association of Thailand(泰国人工智能协会);School of Biomedical Engineering & Imaging Sciences, King’s College London(生物医学工程与成像科学学院,伦敦国王学院);Siriraj Informatics and Data Innovation Center (SIData+), Faculty of Medicine, Siriraj Hospital, Mahidol University(Siriraj信息与数据创新中心(SIData+),医学学院,Siriraj医院,玛希道大学)
本文提出Beyond One World基准,通过30位英雄、90个正典版本及正典事件与道德困境任务,评估LLM跨多元宇宙角色一致性,并以Think-Act Matching衡量推理与行动对齐。
FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Shanghai Institute of AI for Education, East China Normal University(华东师范大学上海人工智能教育研究院);PICC AMC(中国人民保险资产管理公司);Bytedance(字节跳动)
FinBoardBench: Benchmarking Dynamic Wealth Management and Strategic Financial Reasoning of LLMs via Board Game Simulations
School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China(1 计算机科学与工程学院,澳门科技大学,澳门,中国);School of Economics, Anhui University, Anhui, China(2 经济学院,安徽大学,安徽,中国);SKLPlanets, Macau University of Science and Technology, Macau, China(3 SKLPlanets,澳门科技大学,澳门,中国);Department of Computer and Information Science, University of Macau, Macau, China(4 计算机与信息科学系,澳门大学,澳门,中国);School of Energy and Power Engineering, Huazhong University of Science and Technology, Hubei, China(5 能源与动力工程学院,华中科技大学,湖北,中国)
FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information
Georgia Institute of Technology(佐治亚理工学院);Columbia University(哥伦比亚大学);California State University(加州州立大学);University of Montreal(蒙特利尔大学);Carnegie Mellon University(卡内基梅隆大学);Rensselaer Polytechnic Institute(莱斯利理工学院);The University of Manchester(曼彻斯特大学);Harvard University(哈佛大学)
XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning
School of Computing and Information Systems, Singapore Management University(计算与信息系统学院,新加坡管理大学);Institute of Trustworthy Embodied AI, Fudan University(可信具身人工智能研究院,复旦大学)
Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies
Department of Computer Science and Engineering(计算机科学与工程系);New York University(纽约大学);New York, NY 10012(纽约市NY 10012)
FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets
University of California, Los Angeles(加州大学洛杉矶分校);NYU Shanghai(上海纽约大学);Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning(上海人工智能与深度学习前沿科学中心);Columbia University(哥伦比亚大学)
The Automated but Risky Game: Modeling and Benchmarking Agent-to-Agent Negotiations and Transactions in Consumer Markets
Stanford University(斯坦福大学);University of Toronto(多伦多大学);Google DeepMind(谷歌DeepMind);University of Southern California(南加州大学);Massachusetts Institute of Technology(麻省理工学院)
Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection
The University of Manchester(曼彻斯特大学);Stevens Institute of Technology(史蒂文斯理工学院);The Fin AI(Fin AI);Columbia University(哥伦比亚大学);Islamic University of Technology(伊斯兰科技大学);Athens University of Economics and Business(雅典经济与商业大学);Archimedes, Athena Research Center(阿基米德,亚特兰蒂斯研究中心);MBZUAI;McGill University(麦吉尔大学);Mila - Quebec AI Institute(魁北克人工智能研究所);Dubai Police(迪拜警察);University of Melbourne(墨尔本大学);Halmstad University(哈姆斯塔德大学);University of Florida(佛罗里达大学);ELLIS Manchester(曼彻斯特ELLIS)
Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding
Duke University Medical Center(杜克大学医学中心);Patient Revenue Management Organization, Duke University Health System(杜克大学卫生系统患者收入管理组织);Duke Pratt School of Engineering(杜克普拉特工程学院);Duke University School of Medicine(杜克大学医学院);MSTAT