面向语言学习的大语言模型的口语语言智能
Spoken Language Intelligence of Large Language Models for Language Learning
AI总结：
本文通过构建口语语言知识多项选择题数据集并评估20个大语言模型，研究LLMs在语音学、音系学和第二语言习得中的教育应用效果，发现提示技术可显著提升推理性能，但模型在现实问题推理上仍存在局限。
摘要：
人们长期以来一直希望拥有一种能够在现实生活情境中提供辅助的对话系统，而大语言模型（LLMs）的最新进展正使这一设想更接近现实。尽管LLMs在性能上常常令人印象深刻，但它们在需要专家知识的现实场景中的有效性仍不明确。人们认为LLMs在教育领域最具潜力和价值，尤其是在开发基于人工智能（AI）的虚拟教师以促进语言学习方面。我们的研究重点在于评估LLMs在教育领域的有效性，特别是在口语语言学习方面，包括语音学、音系学和第二语言习得。我们引入了一个新的多项选择题数据集，用于评估LLMs在上述场景中的有效性，包括对口语语言知识的理解和应用。此外，我们研究了各种提示技术的影响，例如零样本和少样本方法（在问题前添加问答示例）、思维链（CoT，逐步思考）、领域内示例以及外部工具（Google、Wikipedia）。我们使用这些方法对流行的LLMs（20个不同的模型）进行了大规模评估。与零样本基线相比，我们在实际问题的推理上取得了显著的性能提升（GPT-3.5，49.1% -> 63.1%；LLaMA2-70B-Chat，42.2% -> 48.6%）。我们发现不同规模的模型对语音学、音系学和第二语言习得中的概念有良好的理解，但在现实世界问题的推理方面表现出局限性。此外，我们还探索了关于对话交流的初步发现。
英文摘要：
People have long hoped for a conversational system that can assist in real-life situations, and recent progress on large language models (LLMs) is bringing this idea closer to reality. While LLMs are often impressive in performance, their efficacy in real-world scenarios that demand expert knowledge remains unclear. LLMs are believed to hold the most potential and value in education, especially in the development of Artificial intelligence (AI) based virtual teachers capable of facilitating language learning. Our focus is centered on evaluating the efficacy of LLMs in the realm of education, specifically in the areas of spoken language learning which encompass phonetics, phonology, and second language acquisition. We introduce a new multiple-choice question dataset to evaluate the effectiveness of LLMs in the aforementioned scenarios, including understanding and application of spoken language knowledge. In addition, we investigate the influence of various prompting techniques such as zero- and few-shot method (prepending the question with question-answer exemplars), chain-of-thought (CoT, think step-by-step), in-domain exampler and external tools (Google, Wikipedia). We conducted large-scale evaluation on popular LLMs (20 distinct models) using these methods. We achieved significant performance improvements compared to the zero-shot baseline in the practical questions reasoning (GPT-3.5, 49.1% -> 63.1%; LLaMA2-70B-Chat, 42.2% -> 48.6%). We found that models of different sizes have good understanding of concepts in phonetics, phonology, and second language acquisition, but show limitations in reasoning for real-world problems. Additionally, we also explore preliminary findings on conversational communication.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
