熟能生巧（未必）：回顾微架构侧信道攻击的基准测试实践
Practice Makes (Im)Perfect: A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks
- Univ. Lille, CNRS, Inria（里尔大学，法国国家信息与自动化研究所）
- Univ. Rennes, CNRS, IRISA（雷恩大学，法国国家信息与自动化研究所）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
本文针对微架构侧信道攻击的基准测试核心问题，调查83篇顶级会议论文后识别19种基准测试缺陷，提出评估新攻击的关键属性，揭示了该领域基准测试实践的问题及会议间的差异。
中文摘要 AI 辅助
近年来，微架构侧信道研究以惊人的速度发展，对严谨且有意义的基准测试的需求日益增长。早期的攻击论文通常依赖于间接代理指标，例如 covert-channel（隐蔽信道）带宽或对 naive AES（基础版高级加密标准）、RSA（非对称加密算法）实现的密钥恢复，这些做法形成了事实上的标准，后续许多研究工作继续沿用，有时甚至直接与前人工作的原始数值进行比较。尽管这些实践提供了便捷的比较基准，但当前的基准测试可能并非评估新原语特定属性的最相关指标。更成问题的是，微架构攻击对实验条件极为敏感：目标系统的微小变化就可能显著改变结果和性能。因此，不当的评估实践会破坏可复现性，并使比较的相关性受到质疑，即便在本应识别此类问题的顶级 venues（会议/场所）中也是如此。本文解决了微架构侧信道攻击的适当基准测试这一核心问题，并考察其对该领域研究质量的更广泛影响。我们调查了2014年至2024年间发表在顶级安全与架构会议上的83篇攻击论文，从该语料库中识别并定义了19种反复出现的基准测试缺陷，这些缺陷影响评估的完整性、相关性、合理性和可复现性，包括不公平或缺失的比较、缺少代码或材料、未能评估关键攻击属性等。平均而言，每篇论文存在5.5个此类缺陷，凸显了该问题的普遍性，即便在高度精选的会议中亦是如此。基于我们的发现，我们确定并提出了适当评估新攻击所需的关键属性，还强调了随时间的变化趋势以及安全会议与架构会议之间的不同实践。
英文摘要
Microarchitectural side-channel research has grown at an exceptional pace in recent years, increasing the need for rigorous and meaningful benchmarking. Early attack papers typically relied on indirect proxies, such as covert-channel bandwidth or key-recovery on naive AES and RSA implementations, setting de facto standards that many subsequent works continued to replicate, sometimes by directly comparing against raw numbers from prior work. While these practices offer convenient points of comparison, current benchmarks may not be the most relevant to assess specific properties of new primitives. Even more problematic, microarchitectural attacks are notoriously sensitive to experimental conditions: minimal changes in the target system can significantly alter outcomes and performance. As a result, inadequate evaluation practices undermine reproducibility and cast doubt on the relevance of comparisons, even in top-tier venues where such issues should be identified. This paper tackles the core problem of proper benchmarking for microarchitectural side-channel attacks and examines its broader impact on research quality in the field. We survey 83 attack papers published in top-ranked security and architecture conferences from 2014 to 2024. From this corpus, we identify and define 19 recurrent benchmarking flaws that affect evaluation completeness, relevance, soundness, and reproducibility. These flaws include unfair or absent comparisons, missing code or materials, and the failure to evaluate the key attack properties. On average, each paper exhibits 5.5 such flaws, highlighting how widespread the issue is, even in highly selective venues. Based on our findings, we identify and suggest key properties that are relevant to properly evaluate new attacks. We also highlight trends over time and different practices between security and architecture conferences.
