arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.08070cs.AI
收藏

VecCISC:通过推理轨迹聚类和候选答案选择改进置信度引导的自一致性

VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection

Computer Science Department, Brandeis University(布拉德雷大学计算机科学系)

James Petullo, Sonny George, Dylan Cashman, Nianwen Xue

AI总结:

VecCISC通过推理轨迹聚类和候选答案筛选优化置信度引导的自一致性方法,减少计算开销并保持高精度。

摘要:

一种标准的扩展推理时间推理技术是自一致性,其中从LLM中采样多个候选答案,然后选择最常见的答案。最近研究表明,加权多数投票(如置信度引导的自一致性(CISC))通过为每个候选答案分配置信度值并选择累积分数最高的答案,在广泛流行的基准上更准确。在实践中,加权多数投票需要为每个候选人的推理轨迹调用批评LLM以生成答案的置信度评分。这一二次LLM调用显著增加了加权多数投票的开销和成本,尽管其潜在性能优势。为减少此费用,我们提出了VecCISC,一种轻量级、自适应的框架,利用语义相似性度量来过滤语义等价、退化或虚构的推理轨迹,从而减少需由批评者评估的候选答案数量。为确保充分的实验彻底性,我们在五个具有挑战性的、广泛采用的数据集上评估VecCISC,涵盖数学、化学、生物学、常识推理和人文学科领域。我们的结果表明,VecCISC将总token使用量减少了47%,同时保持或超过了CISC的准确性。

英文摘要:

A standard technique for scaling inference-time reasoning is Self-Consistency, whereby multiple candidate answers are sampled from an LLM and the most common answer is selected. More recently, it has been shown that weighted majority voting (e.g. Confidence-Informed Self Consistency (CISC)), which assigns a confidence value to each candidate answer and chooses the answer with the largest accumulated score, tends to be more accurate on a wide range of popular benchmarks. In practice, weighted majority voting necessitates calling a critic LLM on each candidate's reasoning trace to produce the answer's confidence score. This secondary series of LLM calls greatly increases the overhead and cost of weighted majority voting, despite its potential performance benefits. To reduce this expense, we propose VecCISC, a lightweight, adaptive framework that uses a measure of semantic similarity to filter reasoning traces that are semantically equivalent to others, degenerate, or hallucinated, thus decreasing the number of candidate answers that must be evaluated by the critic. To ensure adequate experimental thoroughness, we evaluate VecCISC on five challenging, widely-adopted datasets spanning the domains of mathematics, chemistry, biology, commonsense reasoning, and the humanities. Our results demonstrate that VecCISC reduces the total token usage by 47%, while maintaining or exceeding the accuracy of CISC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑