UMBCLU 在 SemEval-2024 任务 1A 和 1C 中的表现：结合与不结合机器翻译的语义文本相关性研究
UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation
AI总结：
本研究针对SemEval-2024任务1的语义文本相关性任务，结合多类表现优异的大语言模型构建TranSem和FineSem模型，探究训练方法与机器翻译的作用，在监督和跨语言设置下均取得部分优于基线的成绩。
摘要：
SemEval-2024 任务 1“面向非洲和亚洲语言的语义文本相关性”的目标是开发模型，用于识别多种语言（14 种非洲和亚洲语言）及不同设置（监督、无监督和跨语言）下两个句子之间的语义文本相关性（STR）。大语言模型（LLMs）已在多项自然语言理解任务中展现出出色性能，例如多语言机器翻译（MMT）、语义相似度（STS）以及句子嵌入编码。我们结合在这些任务上表现优异的 LLM，针对监督和跨语言设置开发了两种 STR 模型——$\textit{TranSem}$ 和 $\textit{FineSem}$。我们探究了多种训练方法的有效性以及机器翻译的实用性。研究发现，针对该任务的直接微调效果与使用句子嵌入的效果相当，而将文本翻译成英语能提升部分语言的性能。在监督设置下，我们的模型在 3 种语言上的表现优于官方基线，其余 4 种语言的表现与基线持平。在跨语言设置下，我们的模型在 3 种语言上的表现优于基线（其中南非语获得第 1 名，印度尼西亚语获得第 2 名），2 种语言的表现与基线持平，其余 7 种语言的表现较差。我们的代码已公开在 https://github.com/dipta007/SemEval24-Task8。
英文摘要：
The aim of SemEval-2024 Task 1, "Semantic Textual Relatedness for African and Asian Languages" is to develop models for identifying semantic textual relatedness (STR) between two sentences using multiple languages (14 African and Asian languages) and settings (supervised, unsupervised, and cross-lingual). Large language models (LLMs) have shown impressive performance on several natural language understanding tasks such as multilingual machine translation (MMT), semantic similarity (STS), and encoding sentence embeddings. Using a combination of LLMs that perform well on these tasks, we developed two STR models, $\textit{TranSem}$ and $\textit{FineSem}$, for the supervised and cross-lingual settings. We explore the effectiveness of several training methods and the usefulness of machine translation. We find that direct fine-tuning on the task is comparable to using sentence embeddings and translating to English leads to better performance for some languages. In the supervised setting, our model performance is better than the official baseline for 3 languages with the remaining 4 performing on par. In the cross-lingual setting, our model performance is better than the baseline for 3 languages (leading to $1^{st}$ place for Africaans and $2^{nd}$ place for Indonesian), is on par for 2 languages and performs poorly on the remaining 7 languages. Our code is publicly available at https://github.com/dipta007/SemEval24-Task8.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
