When using language models (LMs) to solve complex problems, humans might struggle to understand the LM-generated solutions and repair the flawed ones. To assist humans in repairing them, we propose to automatically decompose complex solutions into multiple simpler pieces that correspond to specific subtasks. We introduce a novel objective for learning task decomposition, termed assistive value (AssistV), which measures the feasibility and speed for humans to repair the decomposed solution. We collect a dataset of human repair experiences on different decomposed solutions. Utilizing the collected data as in-context examples, we then learn to critique, refine, and rank decomposed solutions to improve AssistV. We validate our method under competitive programming problems: under 177 hours of human study, our method enables non-experts to solve 33.3\% more problems, speeds them up by 3.3x, and empowers them to match unassisted experts.
AutoCode: LLMs as Problem Setters for Competitive Programming
University of California San Diego（加州大学圣地亚哥分校）；New York University（纽约大学）；University of Washington（华盛顿大学）；Princeton University（普林斯顿大学）；University of California Berkeley（加州大学伯克利分校）；OpenAI；Massachusetts Institute of Technology（麻省理工学院）；University of Waterloo（滑铁卢大学）；Sentient Labs
InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
School of Cyber Science and Engineering, Huazhong University of Science and Technology（华中科技大学网络空间科学与工程学院）；Key Laboratory of Cyberspace Security, Ministry of Education（教育部网络空间安全重点实验室）；Hubei Key Laboratory of Distributed System Security（湖北省分布式系统安全重点实验室）
SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving
The University of Hong Kong（香港大学）；University of California Los Angeles（加州大学洛杉矶分校）；Tsinghua University（清华大学）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；The Ohio State University（俄亥俄州立大学）；University of Edinburgh（爱丁堡大学）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；LMSYS Org（LMSYS组织）
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
New York University（纽约大学）；Princeton University（普林斯顿大学）；University of California San Diego（加利福尼亚大学圣地亚哥分校）；University of Washington（华盛顿大学）；Canyon Crest Academy（峡谷克雷斯特学院）；University of Waterloo（滑铁卢大学）；McGill University（麦吉尔大学）
