arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.01322cs.LGstat.ML
收藏

FEET:一种评估嵌入技术的框架

FEET: A Framework for Evaluating Embedding Techniques

UCLA(加利福尼亚大学洛杉矶分校)

Simon A. Lee, John Lee, Jeffrey N. Chiang

AI总结:

本文提出FEET标准化协议,通过冻结、少样本和微调三种嵌入场景及情感分析和医学案例,全面评估基础模型性能,为表示学习研究提供基准。

摘要:

在本研究中,我们引入了FEET,一个旨在指导基础模型开发和基准测试的标准化协议。尽管存在大量用于评估这些模型的基准数据集,我们提出了一个在三种不同场景下的结构化评估协议,以全面了解其实际性能。我们定义了三个主要用例:冻结嵌入、少样本嵌入和完全微调嵌入。每个场景都通过两个案例研究进行了详细说明和展示:一个在情感分析领域,另一个在医学领域,展示了这些评估如何对基础模型在研究应用中的有效性进行彻底评估。我们推荐将此协议作为未来旨在推进表示学习模型研究的标准。

英文摘要:

In this study, we introduce FEET, a standardized protocol designed to guide the development and benchmarking of foundation models. While numerous benchmark datasets exist for evaluating these models, we propose a structured evaluation protocol across three distinct scenarios to gain a comprehensive understanding of their practical performance. We define three primary use cases: frozen embeddings, few-shot embeddings, and fully fine-tuned embeddings. Each scenario is detailed and illustrated through two case studies: one in sentiment analysis and another in the medical domain, demonstrating how these evaluations provide a thorough assessment of foundation models' effectiveness in research applications. We recommend this protocol as a standard for future research aimed at advancing representation learning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑