探索BERT和GPT类大语言模型中的反向诅咒及其他演绎逻辑推理
Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
AI总结：
本研究发现BERT对反向诅咒免疫，而GPT在处理多集合逻辑推理时存在困难，揭示了两种模型在简单与复杂逻辑推理中的差异。
摘要：
反向诅咒一词指的是自回归解码器大语言模型（如ChatGPT）在训练时遇到"A是B"却无法学习"B是A"的情况，假设B和A是不同的且可以相互唯一识别，这表明了基本的逻辑推断失败。这在使用GPT模型进行某些通用任务如构建知识图谱时提出了警示，因为它们遵循这一对称原理。在本研究中，我们检查了双向LLM BERT，并发现其对反向诅咒具有免疫力。受持续利用LLM构建生物医学知识图谱的努力驱动，我们还开始评估更复杂但关键的演绎推理能力。这一过程包括首先训练编码器和解码器语言模型以掌握两个集合的交集和并集操作，然后评估它们对三个新创建集合的不同并集和交集组合操作的推理能力。研究结果表明，虽然编码器和解码器语言模型在涉及两个集合（并集/交集）的任务中表现娴熟，但在涉及三个集合（各种并集和交集组合）的操作时却遇到困难。我们的研究突显了编码器和解码器模型在简单和复杂逻辑推理中的不同特性。在实践中，BERT和GPT之间的选择应根据特定任务的要求和性质进行指导，利用它们在双向上下文理解和序列预测方面的各自优势。
英文摘要：
The term "Reversal Curse" refers to the scenario where auto-regressive decoder large language models (LLMs), such as ChatGPT, trained on "A is B" fail to learn "B is A," assuming that B and A are distinct and can be uniquely identified from each other, demonstrating a basic failure of logical deduction. This raises a red flag in the use of GPT models for certain general tasks such as constructing knowledge graphs, considering their adherence to this symmetric principle. In our study, we examined a bidirectional LLM, BERT, and found that it is immune to the reversal curse. Driven by ongoing efforts to construct biomedical knowledge graphs with LLMs, we also embarked on evaluating more complex but essential deductive reasoning capabilities. This process included first training encoder and decoder language models to master the intersection and union operations on two sets and then moving on to assess their capability to infer different combinations of union and intersection operations on three newly created sets. The findings showed that while both encoder and decoder language models, trained for tasks involving two sets (union/intersection), were proficient in such scenarios, they encountered difficulties when dealing with operations that included three sets (various combinations of union and intersection). Our research highlights the distinct characteristics of encoder and decoder models in simple and complex logical reasoning. In practice, the choice between BERT and GPT should be guided by the specific requirements and nature of the task at hand, leveraging their respective strengths in bidirectional context comprehension and sequence prediction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
