arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.11006cs.CLcs.AIcs.LGcs.PF
收藏

评估基础模型的有效性:推进基准测试实践以增强微调决策

Evaluating the Efficacy of Foundational Models: Advancing Benchmarking Practices to Enhance Fine-Tuning Decision-Making

Rochester Institute of Technology(罗切斯特理工学院)

Oluyemi Enoch Amujo, Shanchieh Jay Yang

AI总结:

本研究评估Gemma-2B和Gemma-7B在网络安全、医学和金融等领域的表现,提出ThroughCut离群点检测技术,发现模型大小和提示类型显著影响响应质量,强调全面基准测试框架的重要性。

摘要:

近年来,大型语言模型(LLMs)已扩展到各个领域。然而,仍然需要评估这些模型在面对常见查询与领域特定查询时的表现,这可能有助于在针对领域特定下游任务进行微调之前进行基准测试。本研究评估了LLMs,特别是Gemma-2B和Gemma-7B,在包括网络安全、医学和金融在内的多个领域中的表现,并与常见知识查询进行了比较。本研究采用了一种全面的方法来评估基础模型,包括问题制定、数据分析以及开发ThroughCut,一种新颖的离群点检测技术,该技术根据响应的简洁性自动识别响应吞吐量的离群点。这种方法论的严谨性增强了所提出的评估框架的可信度。本研究重点评估了推理时间、响应长度、吞吐量、质量和资源利用率,并调查了这些因素之间的相关性。结果表明,模型大小和用于推理的提示类型显著影响了响应长度和质量。此外,包含各种类型查询的常见提示会在不规则的间隔内生成多样且不一致的响应。相比之下,领域特定的提示在合理时间内一致地生成简洁的响应。总体而言,本研究强调了需要全面的评估框架,以增强多领域AI研究中基准测试程序的可靠性。

英文摘要:

Recently, large language models (LLMs) have expanded into various domains. However, there remains a need to evaluate how these models perform when prompted with commonplace queries compared to domain-specific queries, which may be useful for benchmarking prior to fine-tuning for domain-specific downstream tasks. This study evaluates LLMs, specifically Gemma-2B and Gemma-7B, across diverse domains, including cybersecurity, medicine, and finance, compared to common knowledge queries. This study utilizes a comprehensive methodology to assess foundational models, which includes problem formulation, data analysis, and the development of ThroughCut, a novel outlier detection technique that automatically identifies response throughput outliers based on their conciseness. This methodological rigor enhances the credibility of the presented evaluation frameworks. This study focused on assessing inference time, response length, throughput, quality, and resource utilization and investigated the correlations between these factors. The results indicate that model size and types of prompts used for inference significantly influenced response length and quality. In addition, common prompts, which include various types of queries, generate diverse and inconsistent responses at irregular intervals. In contrast, domain-specific prompts consistently generate concise responses within a reasonable time. Overall, this study underscores the need for comprehensive evaluation frameworks to enhance the reliability of benchmarking procedures in multidomain AI research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑