当尺子破损时:基于LLM的安全日志评估中的解析诱导抑制
When the Ruler is Broken: Parsing-Induced Suppression in LLM-Based Security Log Evaluation
AI总结:
本文揭示了基于LLM的安全日志分类器评估中因严格正则表达式解析导致的系统性误差,通过OpenSOC-AI案例显示,修正后的模糊解析可将威胁准确性从0%提升至76%,并提出SOC-Bench v0基准框架以避免解析偏差。
摘要:
基于LLM的SOC日志分类器通常通过正则表达式管道提取自由形式模型输出的结构化字段进行评估。我们证明这种做法引入了一类无声的系统性评估误差,称为解析诱导抑制,可能导致完全功能的模型显得完全不功能。使用OpenSOC-AI,一个经过LoRA微调的TinyLlama-1.1B系统作为可复现案例研究,我们展示严格正则表达式解析器在相同模型输出和评估集上报告了0%的威胁准确性,而修正后的模糊解析器在相同条件下恢复了76%的威胁准确性。76个百分点的差距完全归因于评估方法。严重准确性在两种解析器下保持恒定在58%,提供了一个内置的控制,将字段名称格式不匹配作为因果机制而非模型退化。作为外部参考,Claude Sonnet在相同50个示例集上零样本评估,按照相同模糊协议达到了88%的威胁准确性和58%的严重准确性。模糊评估下的残余误差集中在三个类别中,包括侦察、暴力破解和凭证 stuffing,每个类别贡献了所有4个误分类,这种模式反映了行为上相邻的日志类型之间的类别边界难度,而不是全局模型失败。我们提出SOC-Bench v0,一个包含标准化13类威胁分类法、最低统计功效要求、模糊字段提取规范和公开评分脚本的基准框架,旨在防止未来SOC LLM研究中的解析特定准确性偏差。
英文摘要:
LLM-based SOC log classifiers are commonly evaluated using regular-expression pipelines that extract structured fields from free-form model output. We demonstrate that this practice introduces a class of silent, systematic evaluation errors, which we term parsing-induced suppression that can cause a fully functional model to appear completely non-functional. Using OpenSOC-AI, a LoRA fine-tuned TinyLlama-1.1B system for security log threat classification, as a reproducible case study, we show that a strict regex parser reported 0% threat accuracy while a corrected fuzzy parser recovered 76% threat accuracy on the same model outputs and the same evaluation set. A gap of 76 percentage points attributable entirely to evaluation methodology. Severity accuracy remained constant at 58% under both parsers, providing a built-in control that isolates field name format mismatch as the causal mechanism rather than model degradation. For external reference, Claude Sonnet evaluated zero-shot on the same 50 example set achieved 88% threat accuracy and 58% severity accuracy under the same fuzzy protocol. Residual errors under fuzzy evaluation concentrate in three categories including reconnaissance, brute force, and credential stuffing, each contributing all 4 misclassifications, a pattern that reflects class-boundary difficulty among behaviorally adjacent log types rather than global model failure. We propose SOC-Bench v0, a benchmark framework comprising a standardized 13 category threat taxonomy, minimum statistical power requirements, fuzzy field extraction specification, and a public scoring script intended to prevent parser specific accuracy distortion in future SOC LLM research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。