Recently, large language models (LLMs) have expanded into various domains. However, there remains a need to evaluate how these models perform when prompted with commonplace queries compared to domain-specific queries, which may be useful for benchmarking prior to fine-tuning for domain-specific downstream tasks. This study evaluates LLMs, specifically Gemma-2B and Gemma-7B, across diverse domains, including cybersecurity, medicine, and finance, compared to common knowledge queries. This study utilizes a comprehensive methodology to assess foundational models, which includes problem formulation, data analysis, and the development of ThroughCut, a novel outlier detection technique that automatically identifies response throughput outliers based on their conciseness. This methodological rigor enhances the credibility of the presented evaluation frameworks. This study focused on assessing inference time, response length, throughput, quality, and resource utilization and investigated the correlations between these factors. The results indicate that model size and types of prompts used for inference significantly influenced response length and quality. In addition, common prompts, which include various types of queries, generate diverse and inconsistent responses at irregular intervals. In contrast, domain-specific prompts consistently generate concise responses within a reasonable time. Overall, this study underscores the need for comprehensive evaluation frameworks to enhance the reliability of benchmarking procedures in multidomain AI research.
Boosting Revisited: Benchmarking and Advancing LP-Based Ensemble Methods
Industrial Engineering and Management Science University of Twente(特文特大学工业工程与管理科学学院);CIRRELT & SCALE-AI Chair in Data-Driven Supply Chains Polytechnique Montréal(蒙特利尔大学数据驱动供应链联合主席);LAAS-CNRS Université de Toulouse(图卢兹大学CNRS LAAS研究所)
OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences
University of Science and Technology of China(中国科学技术大学);Institute of Theoretical Physics, Chinese Academy of Sciences(中国科学院理论物理研究所);Hefei National Laboratory(合肥国家实验室);Hefei National Research Center for Physical Sciences at the Microscale(合肥微尺度物质科学国家研究中心);School of Physical Sciences, University of Science and Technology of China(中国科学技术大学物理学院);Institute for Advanced Algorithms Research(先进算法研究院);Endless Frontier(无尽前沿(Endless Frontier))
School of Computer Science and Engineering, Northeastern University, Shenyang, China(计算机科学与工程学院,东北大学,沈阳,中国);NiuTrans Research(NiuTrans研究);Kunming University of Science and Technology(昆明理工大学)
Pet-Bench: Benchmarking the Abilities of Large Language Models as E-Pets in Social Network Services
Fudan University(复旦大学);Xiaohongshu Inc.(小红书公司);Beihang University(北航);Zhejiang University(浙江大学);Singapore University of Technology and Design(新加坡科技设计大学)
FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Shanghai Institute of AI for Education, East China Normal University(华东师范大学上海人工智能教育研究院);PICC AMC(中国人民保险资产管理公司);Bytedance(字节跳动)
Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Huazhong University of Science and Technology(华中科技大学);University of Southern California(南加利福尼亚大学);Midu Technology(米度科技);Northwest University of Political and Law(西北政法大学)
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);Zhipu AI(智谱AI);University of Electronic Science and Technology of China(电子科技大学)
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Tongji University(同济大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tongji University School of Medicine(同济大学医学院);Sun Yat-sen University(中山大学);Fuzhou University(福州大学);Shenzhen Loop Area Institute(深圳河套学院)
CovDocker: Benchmarking Covalent Drug Design with Tasks, Datasets, and Solutions
Huazhong University of Science and Technology(华中科技大学);Microsoft Research AI for Science(微软研究院人工智能科学部);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)