As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g., researchers have declared the death of summarization). In this paper, we use financial report summarization as a case study because financial reports are not only long but also use numbers and tables extensively. We propose a computational framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Cohere. We find that GPT-3.5 and Cohere fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias disappears after shuffling the input for Claude, which suggests that Claude seems to recognize important information. We also conduct a comprehensive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engineering to improve GPT-4's use of numbers with limited success. Overall, our analyses highlight the strong capability of Claude 2 in handling long multimodal inputs compared to GPT-4. The generated summaries and evaluation code are available at https://github.com/ChicagoHAI/characterizing-multimodal-long-form-summarization.
FinAudio: A Benchmark for Audio Large Language Models in Financial Applications
Stevens Institute of Technology（史蒂文斯理工学院）；Columbia University（哥伦比亚大学）；The Fin AI（Fin AI公司）；Domyn（Domyn公司）；Augusta University（奥古斯塔大学）；The University of Manchester（曼彻斯特大学）；University of Montreal（蒙特利尔大学）
Exploring Large Language Models for Financial Applications: Techniques, Performance, and Challenges with FinMA
Lyman Briggs College, Michigan State University（密歇根州立大学Lyman Briggs学院）；Department of Finance, Michigan State University（密歇根州立大学金融系）；African Institute for Mathematical Sciences, Rwanda（刚果(金)数学科学研究所）
MountainLion: A Multi-Modal LLM-Based Agent System for Interpretable and Adaptive Financial Trading
The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Sichuan University（四川大学）；Northwestern University（西北大学）；Indiana University（印第安纳大学）；Emory University（埃默里大学）；Nankai University（南开大学）；MountainLion Research（MountainLion研究机构）；Xi’an University of Electronic Science and Technology（西安电子科技大学）；Kyoto University（京都大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Nanjing university（南京大学）；Tsinghua University（清华大学）；University of Toronto（多伦多大学）
FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysis
National University of Singapore（国立新加坡大学）；Estates Pte Ltd（6Estates公司）；Asian Institute of Digital Finance（亚洲数字金融研究所）；Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）
FinBen: A Holistic Financial Benchmark for Large Language Models
The Fin AI；Wuhan University（武汉大学）；Sichuan University（四川大学）；Columbia University（哥伦比亚大学）；Jiangxi Normal University（江西师范大学）；The University of Manchester（曼彻斯特大学）；Stony Brook University（石溪大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Yunnan University（云南大学）；Nanjing Audit University（南京审计大学）；University of Florida（佛罗里达大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Southwest Jiaotong University（西南交通大学）
Characterizing Financial Market Coverage using Artificial Intelligence
LICEF Research Center, Université TÉLUQ（魁北克大学远程教育大学LICEF研究中心）；LORIA, Université de Lorraine（洛林大学洛林计算机科学及其应用实验室）；Université de Sherbrooke（舍布鲁克大学）；Laplace Insights（拉普拉斯洞察公司）
