AMLworld-Compact:用于成本效益高的大语言模型(LLM)评估与错误诊断的重要性加权下采样方法
AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
AI总结:
针对AMLworld测试集评估LLM成本高的问题,提出AMLworld-Compact评估集,采用HT-Coreset协议和霍维茨-汤普森权重,降低评估成本并提升F1精度,同时发布相关资源。
摘要:
在AMLworld最小测试集上评估大语言模型(LLM)成本高昂,因为超过100万个目标中的每一个都需要单独的图提示。采样可降低成本,但小型平衡子集的未加权精确率和F1值会发生变化。我们提出AMLworld-Compact,这一评估集保留了所有非法测试交易,并从监督得分分层中采样合法交易。其HT-Coreset协议在HI-Small和LI-Small上分别将评估目标减少271倍和611倍。霍维茨-汤普森(Horvitz-Thompson)权重为固定预测提供无偏的全测试混淆计数估计;召回率是精确的,而精确率和F1值是比率估计。在使用图特征预处理器(GFP)特征的保留研究中,一个提升树族为每个配置超过500次抽样的另一个提升树族定义分层。在发布的规模下,F1的均方根误差(RMSE)为0.00-2.58个百分点,而采用HT权重的均匀采样则为4.44-7.41个百分点;保守区间仍然较宽。在评估的图提示下,七个开放权重LLM的加权精确率接近全非法预测基线。在对每个模型一次运行的1000个HI-Small少样本输出的结果分层审计中,词汇检查对84.5%的正确二元判决和56.2%的错误二元判决通过,因此这些标记无法确定正确性。我们在两个https URL发布了评估集、构建代码和审计工具。
英文摘要:
Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。