arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.08808cs.LG
收藏

How2Everything: 从网络挖掘如何操作指南以评估和改进大语言模型

How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs

Allen Institute for AI(艾伦人工智能研究所);University of Maryland(马里兰大学);University of Washington(华盛顿大学)

Yapei Chang, Kyle Lo, Mohit Iyyer, Luca Soldaini

AI总结:

How2Everything通过从网络挖掘操作指南,构建评估集并利用强化学习提升模型性能,实现大规模的能力评估与改进闭环。

摘要:

生成分步骤的

英文摘要:

Generating step-by-step "how-to" procedures is a key LLM capability: how-to advice is commonly requested in chatbots, and step-by-step planning is critical for reasoning over complex tasks. Yet, measuring and improving procedural validity at scale on real-world tasks remains challenging and understudied. To address this, we introduce How2Everything, a scalable framework to evaluate and improve goal-conditioned procedure generation. Our framework includes How2Mine, which mines 351K procedures from 980K web pages across 14 topics and readily scales to larger corpora. From this pool we build How2Bench, a 7K-example evaluation set balanced across topics. To reliably score model outputs, we develop How2Score, an evaluation protocol that uses an LLM judge to detect whether a generation contains any critical failure that would prevent achieving the goal. For low-cost, reproducible evaluation, we distill a frontier model into an open 8B model, achieving 80.5% agreement with human annotators. How2Bench reveals clear scaling trends across model sizes and training stages, providing signal early in pretraining. Finally, RL using How2Score as a reward improves performance on How2Bench by >10 points across three models without systematic regressions on standard benchmarks, with gains robust to superficial source-document memorization or format compliance. Taken together, How2Everything shows how pretraining web data can support a closed loop of capability evaluation and improvement at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑