K-BrowseComp:基于韩国语境的网页浏览代理基准测试
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
AI总结:
针对韩国语境,构建包含400个问题的网页浏览代理基准K-BrowseComp,评估前沿模型性能,发现其准确率显著低于BrowseComp,并公开数据与代码。
摘要:
前沿模型评估正从基础能力(如指令遵循和推理)转向组合性、代理性能力,但韩语代理基准仍然稀缺。我们介绍了K-BrowseComp,一个基于韩国语境的网页浏览代理基准,包含400个问题。其中300个问题的K-BrowseComp-Verified子集由母语为韩语的人手动构建和验证。在该子集上,包括GPT-5.5、DeepSeek-V4-Pro和GLM-5.1在内的前沿LLM仅达到30.00–45.67%,相比BrowseComp大幅下降,而通过韩国专有AI基础模型计划发布的韩语LLM仅获得0.00–10.33%。我们进一步利用解决和创建网页浏览问题之间的不对称性,通过硬样本少样本示例和失败模式导向生成构建了一个100个问题的合成分割。在对抗性过滤的合成诊断分割上,最强模型仅达到26.00%,我们将此分割作为定向压力测试单独报告。我们公开发布了数据和代码。
英文摘要:
Frontier model evaluations are shifting from foundational capabilities (e.g., instruction following and reasoning) toward compositional, agentic ones, but Korean agentic benchmarks remain scarce. We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems. The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers. On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67\%, a substantial drop from BrowseComp, while Korean LLMs released through Korea's Proprietary AI Foundation Model program obtain only 0.00--10.33\%. We further construct a 100-problem synthetic split using hard few-shot exemplars and failure-mode-targeted generation to exploit the asymmetry between solving and creating web browsing problems. On the adversarially filtered synthetic diagnostic split, the strongest model reaches only 26.00\%, and we report this split separately as a targeted stress test. We publicly release our data and code.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。