Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to prohibitively high verification costs and incomplete assessments of a response, making reinforcement learning (RL) post-training with rubric-based rewards difficult to scale. This problem is exacerbated by the fact that often the best way to combine these rubrics into one single reward is also highly prompt-specific. We propose Reinforcement Learning with Adversarial Critic (RLAC), a post-training approach that addresses these challenges via dynamic rubric verification. Our approach employs a large language model (LLM) as a critic that dynamically identifies only the most likely failure modes (e.g., a factual error or unhandled edge case), which are then verified by an external validator to optimize both generator and critic jointly. By training both the generator and the critic, this game enhances the critic's error detection and the generator's output quality while reducing required verifications. Our experiments demonstrate that RLAC improves factual accuracy in text generation and correctness in code generation, while also outperforming exhaustive verification and reward model methods. We show that dynamic critics are more effective than fixed critics, showcasing the potential of RLAC for scaling RL post-training to free-form generation tasks.
Actor-Critic Network for Q&A in an Adversarial Environment
University of Texas at Austin(德克萨斯大学奥斯汀分校)
在Q&A NLP领域已经做了大量的工作,以构建对对抗攻击更鲁棒的模型。两个重点领域是生成对抗性数据,以针对这些情况进行训练,或者修改现有架构以构建鲁棒性。本文介绍了一种方法,将这两个想法结合在一起,训练一个几乎强化学习框架中使用的批评模型。使用对抗SQuAD“Add One Sent”数据集,我们表明这种方法在防止对抗攻击方面有一些有希望的迹象。
针对传统深度学习(DL)系统和算法的对抗攻击已经被广泛研究,并且提出了各种防御。然而,针对深度强化学习(DRL)的此类攻击的可能性和可行性较少探索。由于DRL在各种复杂任务中取得了巨大的成功,设计有效的对抗攻击是构建鲁棒的DRL算法不可或缺的前提。在本文中,我们介绍了两种新的对抗攻击技术\emph{stealthily}和\emph{efficiently}攻击DRL代理。这两种技术使对手能够在最小的关键时刻注入对抗样本,同时对代理造成最严重的损害。第一种技术是\emph{critical point attack}:对手建立一个模型来预测未来的环境状态和代理的行动,评估每种可能的攻击策略的损害,并选择最佳的。第二种技术是\emph{antagonist attack}:对手自动学习一个领域不可知的模型,以发现在一个事件中攻击代理的关键时刻。实验结果证明了我们的技术的有效性。具体来说,为了成功攻击DRL代理,我们的临界点技术只需要1个(TORCS)或2个(Atari Pong和Breakout)步骤,而拮抗剂技术需要不到5个步骤(4个Mujoco任务),这是对最先进方法的重大改进。
A survey on text generation using generative adversarial networks
São Paulo State University(圣保罗州立大学)
本文对使用生成对抗网络进行文本生成的研究进展进行了全面的综述。使用对抗学习来生成文本是有希望的,因为它提供了生成所谓的“自然”语言的替代方案。然而,对抗性文本生成并不是一项简单的任务,因为其最重要的架构生成对抗网络被设计用于处理连续信息(图像)而不是离散数据(文本)。因此,大多数作品都基于三种可能的选择,即,Gumbel-Softmax微分,强化学习和修改的训练目标。所有的替代品在本次调查中进行审查,因为他们提出了最新的方法来生成文本使用基于对抗的技术。所选的作品来自著名的数据库,如Science Direct,IEEEEXplore,Springer,Association for Computing Machinery和arXiv,而每一个选定的工作都经过严格的分析和评估,以呈现其目标,方法和实验结果。
Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation
Department of Information Science, Kyoto University, Kyoto, Japan(京都大学信息科学系);ATR Neural Information Processing Laboratories, Kyoto, Japan(ATR神经信息处理实验室);International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan(神经智能国际研究中心)
From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios
Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich(自主车辆系统教授职位,技术大学慕尼黑工程与设计学院);Munich Institute of Robotics and Machine Intelligence (MIRMI)(慕尼黑机器人与机器智能研究所);TUM School of Computation, Information and Technology, Department of Computer Engineering, Technical University of Munich(技术大学慕尼黑计算、信息与技术学院,计算机工程系)
Robust Driving Control for Autonomous Vehicles: An Intelligent General-sum Constrained Adversarial Reinforcement Learning Approach
Beijing Key Laboratory of Security and Privacy in Intelligent Transportation(北京智能交通安全与隐私重点实验室);Beijing Jiaotong University(北京交通大学);College of Computing and Data Science(计算与数据科学学院);Nanyang Technological University(南洋理工大学);School of Computer Science and Technology(计算机科学与技术学院);Taiyuan University of Technology(太原科技大学);School of Electronics and Information Engineering(电子与信息工程学院)
ARAC: Adaptive Regularized Multi-Agent Soft Actor-Critic in Graph-Structured Adversarial Games
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors
Research Center of Synkrotron, Inc.(同步辐射研究中心公司);Institute for Interdisciplinary Information Core Technology(跨学科核心信息技术研究院);Academy of Advanced Interdisciplinary Research, Xidian University(西安电子科技大学先进交叉学科研究院);Chongqing Changan Automobile Co. Ltd(重庆长安汽车股份有限公司);Synkrotron, Inc.(同步辐射公司)
Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks
Lab of Artificial Intelligence for Education, East China Normal University, Shanghai 200062, China(教育人工智能实验室,东华大学,上海200062,中国);Shanghai Institute of Artificial Intelligence for Education, East China Normal University, Shanghai 200062, China(教育人工智能研究所,东华大学,上海200062,中国);School of Computer Science and Technology, East China Normal University, Shanghai 200062, China(计算机科学与技术学院,东华大学,上海200062,中国)
X-Adv: Physical Adversarial Object Attacks against X-ray Prohibited Item Detection
Beihang University(北京航空航天大学);Zhongguancun Laboratory(中关村实验室);Chinese Academy of Sciences(中国科学院);University of Science and Technology of China(中国科学技术大学);iFLYTEK(科大讯飞);JD Explore Academy(京东探索研究院);Hefei Comprehensive National Science Center(合肥综合性国家科学中心)
Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, CAS(中国科学院自动化研究所);Microsoft Research(微软研究院);City University of Hong Kong(香港城市大学)
LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarios
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)