PhonologyBench：评估大语言模型的音系学能力
PhonologyBench: Evaluating Phonological Skills of Large Language Models
AI总结：
本文提出 PhonologyBench 基准，通过字素到音素转换、音节计数和押韵词生成三个任务评估大语言模型的音系学能力，揭示了其与人类的显著差距并建议按下游应用选择模型。
摘要：
音系学是研究语音结构和发音规则的学科，是大语言模型（LLM）研究中一个关键但常被忽视的组成部分。LLM 被广泛应用于教育工具和诗歌生成等多种利用音系学的下游应用中。此外，LLM 有可能从训练数据中学到不完美的正字法形式与音系形式之间的关联。因此，对 LLM 的音系学能力进行基准测试势在必行。为此，我们提出了 PhonologyBench，这是一个新颖的基准，包含三个诊断任务，旨在明确测试 LLM 在英语中的音系学能力：字素到音素转换、音节计数和押韵词生成。尽管无法访问语音数据，LLM 在 PhonologyBench 任务上仍表现出显著性能。然而，我们观察到在押韵词生成和音节计数上，与人类相比分别存在 17% 和 45% 的显著差距。我们的发现强调了研究 LLM 在音系学任务上表现的重要性，这些任务会无意中影响现实世界的应用。此外，我们鼓励研究人员选择在与下游应用密切相关的音系学任务上表现良好的 LLM，因为我们发现没有一个模型能在所有任务上始终优于其他模型。
英文摘要：
Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
