ChemTEB：化学文本嵌入基准，特定领域嵌入模型性能与效率概览
ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain
AI总结：
ChemTEB是一个专为化学领域设计的文本嵌入基准，通过评估34个模型，揭示现有方法在处理化学信息上的优缺点，为化学NLP应用提供标准化评估框架。
摘要：
近期语言模型的进展开启了信息检索和内容生成的新时代，其中嵌入模型在优化数据表示效率和性能方面发挥着重要作用。尽管诸如大规模文本嵌入基准（MTEB）等基准已经标准化了通用领域嵌入模型的评估，但在化学等专业领域仍存在空白，这些领域由于特定领域的挑战需要量身定制的方法。本文介绍了一个新颖的基准——化学文本嵌入基准（ChemTEB），专门为化学科学设计。ChemTEB解决了化学文献和数据的独特语言和语义复杂性，提供了针对化学领域数据的全面任务套件。通过使用该基准评估34个开源和专有模型，我们揭示了当前方法在处理和理解化学信息方面的优缺点。我们的工作旨在为研究社区提供一个标准化的、特定领域的评估框架，促进为化学相关应用开发更精确和高效的NLP模型。此外，它提供了通用模型在特定领域背景下性能的见解。ChemTEB附带开源代码和数据，进一步增强了其可访问性和实用性。
英文摘要：
Recent advancements in language models have started a new era of superior information retrieval and content generation, with embedding models playing an important role in optimizing data representation efficiency and performance. While benchmarks like the Massive Text Embedding Benchmark (MTEB) have standardized the evaluation of general domain embedding models, a gap remains in specialized fields such as chemistry, which require tailored approaches due to domain-specific challenges. This paper introduces a novel benchmark, the Chemical Text Embedding Benchmark (ChemTEB), designed specifically for the chemical sciences. ChemTEB addresses the unique linguistic and semantic complexities of chemical literature and data, offering a comprehensive suite of tasks on chemical domain data. Through the evaluation of 34 open-source and proprietary models using this benchmark, we illuminate the strengths and weaknesses of current methodologies in processing and understanding chemical information. Our work aims to equip the research community with a standardized, domain-specific evaluation framework, promoting the development of more precise and efficient NLP models for chemistry-related applications. Furthermore, it provides insights into the performance of generic models in a domain-specific context. ChemTEB comes with open-source code and data, contributing further to its accessibility and utility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
