arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.15336cs.AR
收藏

人类-人工智能交互:评估LLM在包含图问题的数字逻辑电路中的推理能力,就设计和分析的创造性而言

Human-AI Interaction: Evaluating LLM Reasoning on Digital Logic Circuit included Graph Problems, in terms of creativity in design and analysis

University of Texas at Dallas(达拉斯大学);University of California Davis(加州大学戴维斯分校)

Yogeswar Reddy Thota, Setareh Rafatirad, Homayoun Houman, Tooraj Nikoubin

AI总结:

评估LLM在数字逻辑电路问题中的推理能力,发现其在复杂问题上与官方答案存在差距,且易陷入教科书模板。

摘要:

大型语言模型(LLMs)越来越多地被本科生用作按需导师,但它们在基于电路和图表的数字逻辑问题上的可靠性仍不明确。我们提出了一项人机研究,评估三种广泛使用的LLM(GPT、Gemini和Claude)在10个本科级别的数字逻辑问题上的表现,这些问题涵盖了非标准计数器、基于JK的状态转换、时序图、频率分割和有限状态机。24名学生进行了成对的模型比较,对每个问题提供了(i)首选模型、(ii)感知正确性、(iii)一致性、(iv)冗余性和(v)信心的判断,同时还提供了对整体模型质量、满意度(例如准确性与清晰度)以及验证答案所需感知的脑力劳动的全球评分。为了基准测试技术有效性,我们对所有十个问题应用了独立评委基于官方解决方案的评估,使用严格的正确性标准。结果揭示了感知有用性和正式正确性之间的持续差距:对于最连续要求高的问题(Q1-Q7),评估的LLM中没有一个能匹配官方答案,尽管它们产生了自信且结构良好的解释,这些解释通常受到学生好评。错误分析表明,模型经常默认使用教科书模板(例如标准级联计数器)并难以将电路结构转换为精确的状态演变和时序行为。这些发现表明,在没有验证支架的情况下,LLM可能在核心数字逻辑主题中不可靠,并可能在本科教学中无意中强化误解。

英文摘要:

Large Language Models (LLMs) are increasingly used by undergraduate students as on-demand tutors, yet their reliability on circuit- and diagram-based digital logic problems remains unclear. We present a human- AI study evaluating three widely used LLMs (GPT, Gemini, and Claude) on 10 undergraduate-level digital logic questions spanning non-standard counters, JK-based state transitions, timing diagrams, frequency division, and finite-state machines. Twenty-four students performed pairwise model comparisons, providing per-question judgments on (i) preferred model, (ii) perceived correctness, (iii) consistency, (iv) verbosity, and (v) confidence, along with global ratings of overall model quality, satisfaction across multiple dimensions (e.g., accuracy and clarity), and perceived mental effort required to verify answers. To benchmark technical validity, we applied an independent judge-based evaluation against official solutions for all ten questions, using strict correctness criteria. Results reveal a consistent gap between perceived helpfulness and formal correctness: for the most sequentially demanding problems (Q1- Q7), none of the evaluated LLMs matched the official answers, despite producing confident, well-structured explanations that students often rated favorably. Error analysis indicates that models frequently default to canonical textbook templates (e.g., standard ripple counters) and struggle to translate circuit structure into exact state evolution and timing behavior. These findings suggest that, without verification scaffolds, LLMs may be unreliable for core digital logic topics and can inadvertently reinforce misconceptions in undergraduate instruction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑