arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.02308cs.CL
收藏

交通灯还是畅通交通?探究大语言模型中的短语语义

Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models

Salesforce Research(Salesforce研究院);University of Pittsburgh(匹兹堡大学)

Rui Meng, Ye Liu, Lifu Tu, Daqing He, Yingbo Zhou, Semih Yavuz

AI总结:

本研究通过三个人工标注数据集考察大语言模型理解短语语义的能力,发现其优于传统嵌入方法但与微调方法相比无显著优势,并分析了提示策略效果及错误原因。

摘要:

短语是人类传达语义的基本语言单位。本研究利用三个人工标注数据集,批判性地考察了基于API的大语言模型(LLMs)理解短语语义的能力。我们评估了LLMs在自然语言指令引导下执行短语语义推理任务的表现,并探究了常见提示技术的影响,包括少样本示例和思维链推理。我们的研究结果表明,LLMs在这些数据集上大幅优于传统嵌入方法;然而,与微调方法相比,它们并未表现出显著优势。先进提示策略的有效性表现出差异性。我们进行了详细的错误分析,以解读LLMs在理解短语语义方面所面临的局限性。代码和数据可在https://github.com/memray/llm_phrase_semantics获取。

英文摘要:

Phrases are fundamental linguistic units through which humans convey semantics. This study critically examines the capacity of API-based large language models (LLMs) to comprehend phrase semantics, utilizing three human-annotated datasets. We assess the performance of LLMs in executing phrase semantic reasoning tasks guided by natural language instructions and explore the impact of common prompting techniques, including few-shot demonstrations and Chain-of-Thought reasoning. Our findings reveal that LLMs greatly outperform traditional embedding methods across the datasets; however, they do not show a significant advantage over fine-tuned methods. The effectiveness of advanced prompting strategies shows variability. We conduct detailed error analyses to interpret the limitations faced by LLMs in comprehending phrase semantics. Code and data can be found at https://github.com/memray/llm_phrase_semantics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑