arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.02404cs.CL
收藏

K-BrowseComp:基于韩国语境的网页浏览代理基准测试

K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts

Chung-Ang University(Chung-Ang 大学);KAIST(韩国科学技术院);Seoul National University(首尔国立大学);OnelineAI;NAVER Cloud AI;Carnegie Mellon University(卡内基梅隆大学)
Nahyun Lee, Dongkeun Yoon, Guijin Son, Geewook Kim, Dayoon Ko, Jeonghun Park, Haneul Yoo, Jaewon Cho, Junghun Park, Changyoon Lee, Kyochul Jang, Jaeyeon Kim, Eu… 展开作者

Nahyun Lee, Dongkeun Yoon, Guijin Son, Geewook Kim, Dayoon Ko, Jeonghun Park, Haneul Yoo, Jaewon Cho, Junghun Park, Changyoon Lee, Kyochul Jang, Jaeyeon Kim, Eunsu Kim, Woojin Cho, Seungone Kim

AI总结:

针对韩国语境,构建包含400个问题的网页浏览代理基准K-BrowseComp,评估前沿模型性能,发现其准确率显著低于BrowseComp,并公开数据与代码。

摘要:

前沿模型评估正从基础能力(如指令遵循和推理)转向组合性、代理性能力,但韩语代理基准仍然稀缺。我们介绍了K-BrowseComp,一个基于韩国语境的网页浏览代理基准,包含400个问题。其中300个问题的K-BrowseComp-Verified子集由母语为韩语的人手动构建和验证。在该子集上,包括GPT-5.5、DeepSeek-V4-Pro和GLM-5.1在内的前沿LLM仅达到30.00–45.67%,相比BrowseComp大幅下降,而通过韩国专有AI基础模型计划发布的韩语LLM仅获得0.00–10.33%。我们进一步利用解决和创建网页浏览问题之间的不对称性,通过硬样本少样本示例和失败模式导向生成构建了一个100个问题的合成分割。在对抗性过滤的合成诊断分割上,最强模型仅达到26.00%,我们将此分割作为定向压力测试单独报告。我们公开发布了数据和代码。

英文摘要:

Frontier model evaluations are shifting from foundational capabilities (e.g., instruction following and reasoning) toward compositional, agentic ones, but Korean agentic benchmarks remain scarce. We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems. The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers. On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67\%, a substantial drop from BrowseComp, while Korean LLMs released through Korea's Proprietary AI Foundation Model program obtain only 0.00--10.33\%. We further construct a 100-problem synthetic split using hard few-shot exemplars and failure-mode-targeted generation to exploit the asymmetry between solving and creating web browsing problems. On the adversarially filtered synthetic diagnostic split, the strongest model reaches only 26.00\%, and we report this split separately as a targeted stress test. We publicly release our data and code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑