Despite the strong research interest in document-level Machine Translation (MT), the test sets dedicated to this task are still scarce. The existing test sets mainly cover topics from the general domain and fall short on specialised domains, such as legal and financial. Also, in spite of their document-level aspect, they still follow a sentence-level logic that does not allow for including certain linguistic phenomena such as information reorganisation. In this work, we aim to fill this gap by proposing a novel test set: DOLFIN. The dataset is built from specialised financial documents, and it makes a step towards true document-level MT by abandoning the paradigm of perfectly aligned sentences, presenting data in units of sections rather than sentences. The test set consists of an average of 1950 aligned sections for five language pairs. We present a detailed data collection pipeline that can serve as inspiration for aligning new document-level datasets. We demonstrate the usefulness and quality of this test set by evaluating a number of models. Our results show that the test set is able to discriminate between context-sensitive and context-agnostic models and shows the weaknesses when models fail to accurately translate financial texts. The test set is made public for the community.
Document-Level Numerical Reasoning across Single and Multiple Tables in Financial Reports
Department of Computer Science and Information Engineering, National Taiwan University(国立台湾大学资讯工程学系);FinTech Center, National Taiwan University(国立台湾大学金融科技中心)
DSC-IITISM at FinCausal 2021: Combining POS tagging with Attention-based Contextual Representations for Identifying Causal Relationships in Financial Documents
Indian Institute of Technology (ISM), Dhanbad(印度理工学院(ISM)达巴德分校)
Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);School of Data Science, Fudan University(复旦大学数据科学学院);Ant Group(蚂蚁集团);School of Information and School of Smart Governance, Renmin University of China(中国人民大学信息学院与智慧治理学院)
Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies
Department of Computer Science and Engineering(计算机科学与工程系);New York University(纽约大学);New York, NY 10012(纽约市NY 10012)
FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs
Columbia University(哥伦比亚大学);Stevens Institute of Technology(史蒂文斯理工学院);Rensselaer Polytechnic Institute(拉特格斯理工学院);University of Montreal(蒙特利尔大学);McGill University(麦吉尔大学);MBZUAI(麦吉尔大学人工智能研究所);Cardiff University(卡迪夫大学);The University of Manchester(曼彻斯特大学);Harvard University(哈佛大学)
Enabling and Analyzing How to Efficiently Extract Information from Hybrid Long Documents with LLMs
School of Software & Microelectronics, Peking University(北京大学软件与微电子学院);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);Microsoft(微软公司);University of Technology Sydney(悉尼科技大学)
Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models
Sustainable Hybrid Document-Routed Retrieval for Financial RAG: Resolving the Robustness-Precision Trade-off
organization= School of Engineering, Stanford University , city= Stanford , state= CA , country= USA;organization= Simon Business School, University of Rochester , city= Rochester , state= NY , country= USA;organization= Accounting \& Information Systems, Rutgers University , city= Newark , state= NJ , country= USA
Aspect-Level Obfuscated Sentiment in Thai Financial Disclosures and Its Impact on Abnormal Returns
Department of Linguistics, Faculty of Arts, Chulalongkorn University(语言学系,艺术学院,朱拉隆梭大学);Sasin Graduate Institute of Business Administration, Chulalongkorn University(工商管理研究生院,朱拉隆梭大学)