We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability.
In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.
Benchmarking Large Language Models for Biomedical Relation Extraction
Interdisciplinary School of Doctoral Studies（跨学科博士研究学院）；Faculty of Mathematics and Computer Science, University of Bucharest（布加勒斯特大学数学与计算机科学学院）；HLT Research Center（HLT研究中心）
Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）
Benchmarking local Hebbian learning rules for memory storage and prototype extraction
KTH Royal Institute of Technology, EECS（皇家理工学院）；Stockholm University, Dept. of Mathematics（斯德哥尔摩大学）；KEIM-Institute, Albstadt-Sigmaringen University, Germany（KEIM研究所，阿尔布施塔特-西格马林根大学，德国）；Digital Futures, KTH Royal Institute of Technology, Stockholm（数字未来，皇家理工学院，斯德哥尔摩）
Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies
Department of Computer Science and Engineering（计算机科学与工程系）；New York University（纽约大学）；New York, NY 10012（纽约市NY 10012）
Benchmarking Pathology Foundation Models for Breast Cancer Survival Prediction
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet（卡罗林斯卡研究所医学流行病学与生物统计学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Division of Oncology, Department of Clinical Sciences Lund, Lund University（卢德大学临床科学系肿瘤学系）；Oxford Suzhou Centre for Advanced Research, University of Oxford（牛津大学苏州先进研究中心）
Extraction of Research Objectives, Machine Learning Model Names, and Dataset Names from Academic Papers and Analysis of Their Interrelationships Using LLM and Network Analysis
Aichi Institute of Technology（爱知工业大学）；University of Tokyo（东京大学）；Hokkaido University（北海道大学）；Nagaoka University of Technology（长冈技术科学大学）；Saitama University（埼玉大学）
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China（计算机学院，国家多媒体软件工程技术研究中心和湖北多媒体与网络通信工程重点实验室，武汉大学，中国）；Alibaba Group, Hangzhou, China（阿里巴巴集团，杭州，中国）；Independent Researcher（独立研究者）；Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates（机器学习系，Mohamed bin Zayed人工智能大学，阿拉伯联合酋长国）
Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机科学系）；Salesforce AI Research（Salesforce AI 研究院）；The Hong Kong University of Science and Technology（香港科学与技术大学）；National University of Singapore（新加坡国立大学）；The Education University of Hong Kong（香港教育大学）
LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases
Tsinghua University（清华大学）；Peking University（北京大学）；University of Glasgow（格拉斯哥大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Queen Mary University of London（伦敦玛丽女王大学）；Northeastern University（东北大学）
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition
Fudan University（复旦大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）；University of New South Wales（新南威尔士大学）；National University of Singapore（新加坡国立大学）；Wuhan University（武汉大学）
On Benchmarking Iris Recognition within a Head-mounted Display for AR/VR Application
Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形研究所（IGD））；TU Darmstadt（达姆施塔特工业大学）；The Norwegian Colour and Visual Computing Laboratory, NTNU（挪威色彩与视觉计算实验室，挪威科技大学（NTNU））；Norwegian Biometrics Laboratory, NTNU（挪威生物识别实验室，挪威科技大学（NTNU））；Mathematical and Applied Visual Computing（数学与应用视觉计算（TU Darmstadt））；Mathematical and Applied Visual Computing, TU Darmstadt（数学与应用视觉计算，达姆施塔特工业大学）
EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction
Division of Computer Science, School of Electrical and Computer Engineering（计算机科学系，电气与计算机工程学院）；National Technical University of Athens（雅典技术大学）；Department of Humanities Social Sciences and Law, School of Applied Mathematical and Physical Sciences（人文社会科学与法律系，应用数学与物理科学学院）
