Modern AI agent implementations such as frontier coding agents chain multiple tools at runtime that create a security surface that per-tool guardrails are unable to address, as individually permitted tools can violate organizational policies when composed. We propose the Dynamic Security Control Compositor (DSCC), a two-phase approach to compositional security for multi-tool agent chains. In Phase 1, at session checkout, a Most Restrictive Set (MRS) algorithm composes per-tool security policies into a single effective policy with a formal monotonicity invariant that extending a chain can only tighten the result, blocking incompatible combinations before any tool executes. Outputs of any tool call propagate their classification constraints into a session-level taint state, so subsequent invocations must satisfy the most restrictive constraints seen so far. In Phase 2, at runtime, the system tracks the sensitivity of data the agent touches through a monotonic taint state and revokes the session if the accumulated exposure would make a subsequent tool call a policy violation. Together, these phases provide defense in depth, where static composition prevents unsafe chains from starting, and runtime taint tracking catches violations that emerge from the specific data used. We provide a reference implementation on 32 tools governed by 16 NIST SP 800-53 aligned policies and evaluate it under two composition modes. In the default clearance mode, permitted combinations are partitioned into classification-level clusters, blocking 79.2% of policy pairs and 95.5% of triples. The alternative taint mode admits mixed-classification chains within the exfiltration boundary, blocking 42.5% and 60.5% respectively. We discuss the governance implications for organizations deploying multi-tool agents, including the utility-security tradeoff and the changes needed to operationalize chain-aware policies.
In hierarchical planning for Markov decision processes (MDPs), temporal abstraction allows planning with macro-actions that take place at different time scale in form of sequential composition. In this paper, we propose a novel approach to compositional reasoning and hierarchical planning for MDPs under temporal logic constraints. In addition to sequential composition, we introduce a composition of policies based on generalized logic composition: Given sub-policies for sub-tasks and a new task expressed as logic compositions of subtasks, a semi-optimal policy, which is optimal in planning with…
We propose a novel solution to challenging sparse-reward, continuous control problems that require hierarchical planning at multiple levels of abstraction. Our solution, dubbed AlphaNPI-X, involves three separate stages of learning. First, we use off-policy reinforcement learning algorithms with experience replay to learn a set of atomic goal-conditioned policies, which can be easily repurposed for many tasks. Second, we learn self-models describing the effect of the atomic policies on the environment. Third, the self-models are harnessed to learn recursive compositional programs with multiple…
We study the problem of learning control policies for complex tasks given by logical specifications. Recent approaches automatically generate a reward function from a given specification and use a suitable reinforcement learning algorithm to learn a policy that maximizes the expected reward. These approaches, however, scale poorly to complex tasks that require high-level planning. In this work, we develop a compositional learning approach, called DiRL, that interleaves high-level planning and reinforcement learning. First, DiRL encodes the specification as an abstract graph; intuitively, verti…
Indian Institute of Technology, Madras（印度马德拉斯理工学院）
Deep Reinforcement Learning (DRL) methods have performed well in an increasing numbering of high-dimensional visual decision making domains. Among all such visual decision making problems, those with discrete action spaces often tend to have underlying compositional structure in the said action space. Such action spaces often contain actions such as go left, go up as well as go diagonally up and left (which is a composition of the former two actions). The representations of control policies in such domains have traditionally been modeled without exploiting this inherent compositional structure…
We propose a method for learning expressive energy-based policies for continuous states and actions, which has been feasible only in tabular domains before. We apply our method to learning maximum entropy policies, resulting into a new algorithm, called soft Q-learning, that expresses the optimal policy via a Boltzmann distribution. We use the recently proposed amortized Stein variational gradient descent to learn a stochastic sampling network that approximates samples from this distribution. The benefits of the proposed algorithm include improved exploration and compositionality that allows t…
Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur（计算机科学与工程系，印度理工学院Kharagpur分校）；Department of Mechanical Engineering, Indian Institute of Technology Kharagpur（机械工程系，印度理工学院Kharagpur分校）
University of Pennsylvania（宾夕法尼亚大学）；Brown University（布朗大学）
A hallmark of human intelligence is the ability to construct self-contained chunks of knowledge and reuse them in novel combinations for solving different problems. Learning such compositional structures has been a challenge for artificial systems, due to the underlying combinatorial search. To date, research into compositional learning has largely proceeded separately from work on lifelong or continual learning. This dissertation integrated these two lines of work to present a general-purpose framework for lifelong learning of functionally compositional structures. The framework separates the…
University of California Berkeley（加州大学伯克利分校）；Horizon Robotics（地平线机器人）
The purpose of multi-task reinforcement learning (MTRL) is to train a single policy that can be applied to a set of different tasks. Sharing parameters allows us to take advantage of the similarities among tasks. However, the gaps between contents and difficulties of different tasks bring us challenges on both which tasks should share the parameters and what parameters should be shared, as well as the optimization challenges due to parameter sharing. In this work, we introduce a parameter-compositional approach (PaCo) as an attempt to address these challenges. In this framework, a policy subsp…
Consumer IoT is characterized by heterogeneous devices with diverse functionality and programming interfaces. This lack of homogeneity makes the integration and security management of IoT infrastructures a daunting task for users and administrators. In this paper, we introduce VISCR, a Vendor-Independent policy Specification and Conflict Resolution engine that enables conflict-free policy specification and enforcement in IoT environments. VISCR converts the topology of the IoT infrastructure into a tree-based abstraction and translates existing policies from heterogeneous vendor-specific progr…
As networks expand in size and complexity, they pose greater administrative and management challenges. Software Defined Networks (SDN) offer a promising approach to meeting some of these challenges. In this paper, we propose a policy driven security architecture for securing end to end services across multiple SDN domains. We develop a language based approach to design security policies that are relevant for securing SDN services and communications. We describe the policy language and its use in specifying security policies to control the flow of information in a multi-domain SDN. We demonstra…
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（得克萨斯大学奥斯汀分校奥登计算工程与科学研究所）；Uppsala University（乌普萨拉大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
We propose a framework for verifiable and compositional reinforcement learning (RL) in which a collection of RL subsystems, each of which learns to accomplish a separate subtask, are composed to achieve an overall task. The framework consists of a high-level model, represented as a parametric Markov decision process (pMDP) which is used to plan and to analyze compositions of subsystems, and of the collection of low-level subsystems themselves. By defining interfaces between the subsystems, the framework enables automatic decompositions of task specifications, e.g., reach a target set of states…
Contextual Robotics Institute（情境机器人研究所）；UC San Diego（加州大学圣地亚哥分校）；Open Source Robotics Foundation（开源机器人基金会）
SROS is a proposed addition to the ROS API and ecosystem to support modern cryptography and security measures. An overview of current progress will be presented, rationalizing each major advancement, including: over-the-wire cryptography for all data transport, namespaced access control enforcing graph policies/restrictions, and finally process profiles using Linux Security Modules to harden a node's resource access. By making the community aware of the vulnerabilities in ROS, as well as the proposed solutions provided by SROS, we intend to improve the state of security for future robotics sub…
While data from Supervisory Control And Data Acquisition (SCADA) systems is sent upstream, it is both the length of pulses as well as their frequency present an excellent opportunity to incor-porate statistical fingerprinting. This is so, because datagrams in SCADA traffic follow a poison distribution. Although wrapping the SCADA traffic in a protective IPsec stream is an obvious choice, thin clients and unreliable communication channels make is less than ideal to use crypto-graphic solutions for security SCADA traffic. In this paper, we propose a smart alternative of data obfuscation in the f…
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Schwartz Reisman Institute for Technology and Society（施瓦茨·赖斯曼技术与社会研究所）
We address the problem of teaching a deep reinforcement learning (RL) agent to follow instructions in multi-task environments. Instructions are expressed in a well-known formal language -- linear temporal logic (LTL) -- and can specify a diversity of complex, temporally extended behaviours, including conditionals and alternative realizations. Our proposed learning approach exploits the compositional syntax and the semantics of LTL, enabling our RL agent to learn task-conditioned policies that generalize to new instructions, not observed during training. To reduce the overhead of learning LTL s…
ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL
School of Computing University of Georgia（乔治亚大学计算学院）；School of Computing Sciences & Computer Engineering The University of Southern Mississippi（密西西比大学计算科学与计算机工程学院）
The increasing availability of cloud computing and scientific super computers brings great potential for making R accessible through public or shared resources. This allows us to efficiently run code requiring lots of cycles and memory, or embed R functionality into, e.g., systems and web services. However some important security concerns need to be addressed before this can be put in production. The prime use case in the design of R has always been a single statistician running R on the local machine through the interactive console. Therefore the execution environment of R is entirely unrestr…
The development in the area of wireless communication, mobile and embedded computing leads to significant changes in the application of devices. Over the last years, embedded devices were brought into the consumer area creating the Internet of Things. Furthermore, industrial applications increasingly rely on communication through trust boundaries. Networking is cheap and easily applicable while providing the possibility to make everyday life more easy and comfortable and industry more efficient and less time-consuming. One of the crucial parts of this interconnected world is sound and secure a…
Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework
Hong Kong University of Science and Technology（香港科技大学）；Lingnan University（岭南大学）；ETH Zurich（苏黎世联邦理工学院）；Zhejiang University of Technology（浙江工业大学）；University of Oregon（俄勒冈大学）
