离线强化学习中通过投毒攻击破坏序列级数据-策略覆盖度
Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning
AI总结：
针对离线强化学习现有研究忽略覆盖不足安全风险、单步分析不符多步决策本质的问题，提出序列级可集中系数量化覆盖度，设计CSDPC投毒攻击，仅投毒1%数据即可使智能体性能降90%，为离线RL安全研究提供新视角。
摘要：
离线强化学习（RL）高度依赖预收集数据对目标策略分布的覆盖度。现有研究旨在提升数据-策略覆盖度以缓解分布偏移，却忽略了覆盖不足带来的安全风险，且单步分析与离线RL的多步决策本质不符。为解决该问题，我们引入序列级可集中系数来量化覆盖度，并通过理论分析揭示其对估计误差上界的指数放大效应。在此基础上，我们提出Collapsing Sequence-Level Data-Policy Coverage（CSDPC，破坏序列级数据-策略覆盖度）投毒攻击。考虑到离线RL数据的连续性，我们将状态-动作对转化为决策单元，并提取能捕捉多步行为的代表性决策模式。我们识别出可能导致覆盖不足的罕见模式，并对其投毒以降低覆盖度、加剧分布偏移。实验表明，仅投毒1%的数据集就能使智能体性能下降90%。该发现为分析和保障离线RL的安全性提供了新视角。
英文摘要：
Offline reinforcement learning (RL) heavily relies on the coverage of pre-collected data over the target policy's distribution. Existing studies aim to improve data-policy coverage to mitigate distributional shifts, but overlook security risks from insufficient coverage, and the single-step analysis is not consistent with the multi-step decision-making nature of offline RL. To address this, we introduce the sequence-level concentrability coefficient to quantify coverage, and reveal its exponential amplification on the upper bound of estimation errors through theoretical analysis. Building on this, we propose the Collapsing Sequence-Level Data-Policy Coverage (CSDPC) poisoning attack. Considering the continuous nature of offline RL data, we convert state-action pairs into decision units, and extract representative decision patterns that capture multi-step behavior. We identify rare patterns likely to cause insufficient coverage, and poison them to reduce coverage and exacerbate distributional shifts. Experiments show that poisoning just 1% of the dataset can degrade agent performance by 90%. This finding provides new perspectives for analyzing and safeguarding the security of offline RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
