CircuChain: 解构LLM电路分析中的能力与合规性
CircuChain: Disentangling Competence and Compliance in LLM Circuit Analysis
AI总结：
CircuChain通过设计控制/陷阱问题对，评估LLM在电路分析中遵循指令与物理推理能力的差异，揭示模型能力与约束对齐的不足。
摘要：
随着大型语言模型（LLM）在工程领域接近专家水平，可靠推理在用户指定约束下变得至关重要。在电路分析中，即使数值正确，若违反已确立的方法学惯例如回路方向性或极性分配，也可能在安全关键系统中传播错误。然而，前沿模型是否真正应用基本原理推理还是依赖于训练先验与显式指令冲突仍不清楚。我们引入CircuChain，一个诊断基准，旨在解构电气电路分析中的指令合规性与物理推理能力。CircuChain包含五个经典电路拓扑的控制/陷阱问题对，辅以系统性的符号约定、电流方向和极性定义变化。多阶段验证流程，结合符号求解器、SPICE模拟和基于LLM的错误分类法，可细粒度地将失败归因于惯例错误、物理错误、算术错误或幻觉。在100个任务每模型的测试中，我们观察到一致的合规性-能力分歧。最强模型表现出接近完美的物理推理，但当陷阱条件故意反转自然符号模式时，存在高比例的惯例违规。相反，较弱模型显示较低的物理精度，但更符合显式指令。这些结果表明，模型能力的增加并不保证约束对齐的改进，强调了需要新的评估框架来在数学严格领域强调指令遵循。CircuChain提供了一个这样的框架，并为工程教育和AI对齐研究提供了可操作的见解。
英文摘要：
As large language models (LLMs) advance toward expert-level performance in engineering domains, reliable reasoning under user-specified constraints becomes critical. In circuit analysis, for example, a numerically correct solution is insufficient if it violates established methodological conventions such as mesh directionality or polarity assignments, errors that can propagate in safety-critical systems. Yet it remains unclear whether frontier models truly apply first-principles reasoning or rely on entrenched training priors that conflict with explicit instructions. We introduce CircuChain, a diagnostic benchmark designed to disentangle instruction compliance from physical reasoning competence in electrical circuit analysis. CircuChain consists of counterbalanced Control/Trap problem pairs across five canonical circuit topologies, augmented with systematic variations in sign conventions, current orientations, and polarity definitions. A multi-stage verification pipeline, combining symbolic solvers, SPICE simulation, and an LLM-based error taxonomy, enables fine-grained attribution of failures to convention errors, physics errors, arithmetic mistakes, or hallucinations. Across 100 tasks per model, we observe a consistent Compliance-Competence Divergence. The strongest model evaluated exhibits near-perfect physical reasoning but a high rate of convention violations when Trap conditions deliberately invert natural sign patterns. Conversely, weaker models display lower physical fidelity yet superior adherence to explicit instructions. These results suggest that increased model capability does not guarantee improved constraint alignment and highlight the need for new evaluation frameworks that stress instruction-following under mathematically rigid domains. CircuChain provides one such framework and offers actionable insights for both engineering education and AI alignment research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
