arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.13709cs.CLcs.AI
收藏

列词汇关联(CVA):无数据表的语义解释

Column Vocabulary Association (CVA): semantic interpretation of dataless tables

Vrije Universiteit Amsterdam(阿姆斯特丹自由大学)

Margherita Martorana, Xueli Pan, Benno Kruit, Tobias Kuhn, Jacco van Ossenbruggen

AI总结:

针对仅使用元数据进行语义表解释的新挑战,提出列词汇关联(CVA)任务,评估了多种LLM与RAG组合及SemanticBERT方法,发现LLM在低温度下表现优异,但在数据高度相关时传统方法更具优势。

摘要:

传统的语义表解释(STI)方法主要依赖底层表数据来创建语义注释。今年的SemTab挑战赛引入了“元数据到知识图谱”赛道,重点在于仅使用元数据信息来执行STI,而无需访问底层数据。为应对这一新挑战,我们引入了一个新术语:列词汇关联(CVA)。该术语指仅基于元数据信息对列表头进行语义注释的任务。在本研究中,我们评估了执行CVA任务时多种方法的性能,包括大型语言模型(LLMs)和检索增强生成(RAG)方法,以及一种更传统的基于SemanticBERT的相似度方法。我们的方法采用零样本设置,不向LLMs传递预训练或示例,旨在避免特定领域的设置。我们共调查了7种不同的LLMs,其中三个商业GPT模型(即gpt-3.5-turbo-0.125、gpt-4o和gpt-4-turbo)和四个开源模型(即llama3-80b、llama3-7b、gemma-7b和mixtral-8x7b)。我们将这些模型与RAG系统集成,并探讨了温度设置的变化如何影响性能。此外,我们继续利用SemanticBERT执行CVA任务,分析各种元数据信息如何影响其性能。初步结果表明,LLMs在低于1.0的温度下通常表现良好,在某些情况下准确率达到100%。然而,我们的研究也揭示,数据的性质显著影响CVA任务的结果。事实上,在输入数据和词汇表相关的情况下(例如由同一组织创建),传统方法的性能似乎超越了LLMs。

英文摘要:

Traditional Semantic Table Interpretation (STI) methods rely primarily on the underlying table data to create semantic annotations. This year's SemTab challenge introduced the ``Metadata to KG'' track, which focuses on performing STI by using only metadata information, without access to the underlying data. In response to this new challenge, we introduce a new term: Column Vocabulary Association (CVA). This term refers to the task of semantic annotation of column headers solely based on metadata information. In this study, we evaluate the performance of various methods in executing the CVA task, including a Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) approach, as well as a more traditional similarity approach with SemanticBERT. Our methodology uses a zero-shot setting, with no pretraining or examples passed to the Large Language Models (LLMs), as we aim to avoid a domain-specific setting. We investigate a total of 7 different LLMs, of which three commercial GPT models (i.e. gpt-3.5-turbo-0.125, gpt-4o and gpt-4-turbo) and four open source models (i.e. llama3-80b, llama3-7b, gemma-7b and mixtral-8x7b). We integrate this models with RAG systems, and we explore how variations in temperature settings affect performances. Moreover, we continue our investigation by performing the CVA task utilizing SemanticBERT, analyzing how various metadata information influence its performance. Initial findings indicate that LLMs generally perform well at temperatures below 1.0, achieving an accuracy of 100\% in certain cases. Nevertheless, our investigation also reveal that the nature of the data significantly influences CVA task outcomes. In fact, in cases where the input data and glossary are related (for example by being created by the same organizations) traditional methods appear to surpass the performance of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑