人在回路中的思维链
Human-in-the-Loop through Chain-of-Thought
AI总结：
针对语言模型在复杂推理中的不足，提出基于思维链的人机协同系统MCS及成本效用模型CAMLOP，在12个数据集上验证了其成本与效用的优势。
摘要：
尽管强大语言模型的出现以及思维链提示使得自动化变得越来越普遍，但它在长期或多步逻辑推理中有时会暴露出弱点。例如，在没有人工参与的情况下，用户并不总能获得复杂数学问题的理想答案。在此背景下，我们提出了手动修正系统（MCS）——一个由思维链提示增强的人机协同系统，该系统探索了在推理过程中手动修正子逻辑如何提升LLM的推理性能。更进一步，考虑到一个包含人在回路的系统不仅涉及人类提升性能，还涉及控制成本，因此，我们基于经典经济学理论提出了一个人机协同系统的成本效用分析模型（CAMLOP），用以分析、量化并平衡效用与相应成本。我们使用十二个数据集对MCS和CAMLOP进行了实验。在成本和效用方面的显著优势证明了其优于强基线的优越性。
英文摘要：
While the emergence of powerful language models along with Chain-of-thought prompting has made automation more and more omnipresent, it sometimes demonstrates its weakness in long-term or multi-step logical reasoning. For example, users don't always get desirable answers for complex mathematical problems without human involvement. Against this background, we present the Manual Correction System (MCS) -- a human-in-the-loop system enhanced by Chain-of-Thought prompting, which explores how manual correction of sub-logics in rationales can improve LLM's reasoning performance. Moving one step forward, considering a system with human-in-the-loop involves more than having humans improve performance but also controlling the cost. Therefore, we post a Cost-utility Analysis Model for Human-in-the-Loop systems (CAMLOP) based on classical economics theory to analyze, quantify and balance the utility and the corresponding cost. We conduct experiments of MCS and CAMLOP with twelve datasets. A significant advantage w.r.t cost and utility proves its superiority over strong baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
