聊天机器人接受数学与逻辑问题测试：ChatGPT-3.5、ChatGPT-4 和 Google Bard 的初步比较与评估
Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard
AI总结：
本研究比较了ChatGPT-3.5、ChatGPT-4和Google Bard在数学与逻辑问题上的表现，发现ChatGPT-4最优，Bard在已发表问题上领先，但复杂问题答案可靠性不足。
摘要：
本文对基于大型语言模型的三个聊天机器人，即 ChatGPT-3.5、ChatGPT-4 和 Google Bard，进行了比较，重点关注它们在数学和逻辑问题中给出正确答案的能力。具体而言，我们检验了它们理解当前问题的能力；应用适当算法或方法解决问题的能力；以及生成连贯回应和正确答案的能力。我们使用了30个清晰、无歧义、仅用纯文本完整描述且具有唯一明确正确答案的问题。这些问题被分为两组，每组15个。A组问题为15个“原创”问题，无法在网上找到；而B组包含15个“已发表”问题，可在网上找到，通常附有解答。每个问题对每个聊天机器人提出三次。我们记录并讨论了回答，突出了它们的优点和缺点。研究发现，对于直接的算术、代数表达式或基础逻辑谜题，聊天机器人可能提供准确的解答，尽管并非每次尝试都如此。然而，对于更复杂的数学问题或高级逻辑任务，它们的回答虽然通常以“令人信服”的方式书写，但可能不可靠。一致性问题也值得关注，因为很多时候，当同一问题被多次提出时，聊天机器人会给出相互矛盾的答案。我们通过对三个聊天机器人的最终答案进行正确性评分，进行了比较性的定量评估。结果发现，ChatGPT-4 在两组问题中均优于 ChatGPT-3.5。Bard 在A组原创问题中排名第三，落后于其他两个聊天机器人，而在B组已发表问题中表现最佳（第一名）。这可能是因为 Bard 可以直接访问互联网，而 ChatGPT 聊天机器人则无法与外部世界进行任何通信。
英文摘要：
A comparison between three chatbots which are based on large language models, namely ChatGPT-3.5, ChatGPT-4 and Google Bard is presented, focusing on their ability to give correct answers to mathematics and logic problems. In particular, we check their ability to Understand the problem at hand; Apply appropriate algorithms or methods for its solution; and Generate a coherent response and a correct answer. We use 30 questions that are clear, without any ambiguities, fully described with plain text only, and have a unique, well defined correct answer. The questions are divided into two sets of 15 each. The questions of Set A are 15 "Original" problems that cannot be found online, while Set B contains 15 "Published" problems that one can find online, usually with their solution. Each question is posed three times to each chatbot. The answers are recorded and discussed, highlighting their strengths and weaknesses. It has been found that for straightforward arithmetic, algebraic expressions, or basic logic puzzles, chatbots may provide accurate solutions, although not in every attempt. However, for more complex mathematical problems or advanced logic tasks, their answers, although written in a usually "convincing" way, may not be reliable. Consistency is also an issue, as many times a chatbot will provide conflicting answers when given the same question more than once. A comparative quantitative evaluation of the three chatbots is made through scoring their final answers based on correctness. It was found that ChatGPT-4 outperforms ChatGPT-3.5 in both sets of questions. Bard comes third in the original questions of Set A, behind the other two chatbots, while it has the best performance (first place) in the published questions of Set B. This is probably because Bard has direct access to the internet, in contrast to ChatGPT chatbots which do not have any communication with the outside world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
