Packet routing is one of the fundamental problems in computer networks in which a router determines the next-hop of each packet in the queue to get it as quickly as possible to its destination. Reinforcement learning (RL) has been introduced to design autonomous packet routing policies with local information of stochastic packet arrival and service. However, the curse of dimensionality of RL prohibits the more comprehensive representation of dynamic network states, thus limiting its potential benefit. In this paper, we propose a novel packet routing framework based on \emph{multi-agent} deep reinforcement learning (DRL) in which each router possess an \emph{independent} LSTM recurrent neural network for training and decision making in a \emph{fully distributed} environment. The LSTM recurrent neural network extracts routing features from rich information regarding backlogged packets and past actions, and effectively approximates the value function of Q-learning. We further allow each route to communicate periodically with direct neighbors so that a broader view of network state can be incorporated. Experimental results manifest that our multi-agent DRL policy can strike the delicate balance between congestion-aware and shortest routes, and significantly reduce the packet delivery time in general network topologies compared with its counterparts.
Wesleyan University（卫斯理大学）；University of Connecticut（康涅狄格大学）；Fordham University（福特汉姆大学）
While routing in wireless networks has been studied extensively, existing protocols are typically designed for a specific set of network conditions and so cannot accommodate any drastic changes in those conditions. For instance, protocols designed for connected networks cannot be easily applied to disconnected networks. In this paper, we develop a distributed routing strategy based on deep reinforcement learning that generalizes to diverse traffic patterns, congestion levels, network connectivity, and link dynamics. We make the following key innovations in our design: (i) the use of relational…
University of Southampton（南安普顿大学）；Bournemouth University（伯恩茅斯大学）；Beihang University（北京航空航天大学）
Data packet routing in aeronautical ad-hoc networks (AANETs) is challenging due to their high-dynamic topology. In this paper, we invoke deep reinforcement learning for routing in AANETs aiming at minimizing the end-to-end (E2E) delay. Specifically, a deep Q-network (DQN) is conceived for capturing the relationship between the optimal routing decision and the local geographic information observed by the forwarding node. The DQN is trained in an offline manner based on historical flight data and then stored by each airplane for assisting their routing decisions during flight. To boost the learn…
Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；CA Technologies（CA科技公司）
In this paper we design and evaluate a Deep-Reinforcement Learning agent that optimizes routing. Our agent adapts automatically to current traffic conditions and proposes tailored configurations that attempt to minimize the network delay. Experiments show very promising performance. Moreover, this approach provides important operational advantages with respect to traditional optimization algorithms.
Reinforcement learning means learning a policy--a mapping of observations into actions--based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. We present an application of gradient ascent algorithm for reinforcement learning to a complex domain of packet routing in network communication and compare the performance of this algorithm to other routing methods on a benchmark problem.
Beijing University of Posts and Telecommunications（北京邮电大学）
Routing is one of the key functions for stable operation of network infrastructure. Nowadays, the rapid growth of network traffic volume and changing of service requirements call for more intelligent routing methods than before. Towards this end, we propose a definition of cognitive routing and an implementation approach based on Deep Reinforcement Learning (DRL). To facilitate the research of DRL-based cognitive routing, we introduce a simulator named RL4Net for DRL-based routing algorithm development and simulation. Then, we design and implement a DDPG-based routing algorithm. The simulation…
Value-Decomposed Reinforcement Learning Framework for Taxiway Routing with Hierarchical Conflict-Aware Observations
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University（合成视觉基础科学国家重点实验室，四川大学）；College of Computer Science, Sichuan University（计算机学院，四川大学）
Georgia Institute of Technology（佐治亚理工学院）；IBM Watson AI Foundation Group（IBM沃森人工智能基金会组）
Applying Machine Learning (ML) techniques to design and optimize computer architectures is a promising research direction. Optimizing the runtime performance of a Network-on-Chip (NoC) necessitates a continuous learning framework. In this work, we demonstrate the promise of applying reinforcement learning (RL) to optimize NoC runtime performance. We present three RL-based methods for learning optimal routing algorithms. The experimental results show the algorithms can successfully learn a near-optimal solution across different environment states. Reproducible Code: github.com/huckiyang/interco…
Global routing has been a historically challenging problem in electronic circuit design, where the challenge is to connect a large and arbitrary number of circuit components with wires without violating the design rules for the printed circuit boards or integrated circuits. Similar routing problems also exist in the design of complex hydraulic systems, pipe systems and logistic networks. Existing solutions typically consist of greedy algorithms and hard-coded heuristics. As such, existing approaches suffer from a lack of model flexibility and non-optimum solutions. As an alternative approach, …
Backpressure-based adaptive routing algorithms where each packet is routed along a possibly different path have been extensively studied in the literature. However, such algorithms typically result in poor delay performance and involve high implementation complexity. In this paper, we develop a new adaptive routing algorithm built upon the widely-studied back-pressure algorithm. We decouple the routing and scheduling components of the algorithm by designing a probabilistic routing table which is used to route packets to per-destination queues. The scheduling decisions in the case of wireless n…
This paper is a quantitative analysis on packet switched network with a view to generalize load balancing and determination of appropriate routing algorithm in multipath environment. Several routing algorithms have been introduced for routing of packets from source to destination. Some of them route packets accurately with increased workload and some of them drastically cut down the workload. A few of them can find out a minimum workload deviation for both UDP and TCP packets. We simulated these approaches in a well defined simulator, analyzed and evaluated their performance. After expanding o…
We present a novel geographical routing scheme for spontaneous wireless mesh networks. Greedy geographical routing has many advantages, but suffers from packet losses occurring at the border of voids. In this paper, we propose a flexible greedy routing scheme that can be adapted to any variant of geographical routing and works for any connectivity graph, not necessarily Unit Disk Graphs. The idea is to reactively detect voids, backtrack packets, and propagate information on blocked sectors to reduce packet loss. We also propose an extrapolating algorithm to reduce the latency of void discovery…
Filtering packet traffic and rules of permit/denial of data packets into network nodes are granted by facilitating Access Control Lists (ACL). This paper proposes a procedure of adding a link load threshold value to the access control list rules option, which acts on the basis of threshold value. The ultimate goal of this enhanced ACL is to avoid congestion in targeted subnetworks. The link load threshold value allows to decide that packet traffic is rerouted by the router to avoid congestion, or packet drop happens on the basis of packet priorities. The packet rerouting in case of high traffi…
In this work, a new region-based, multipath-enabled packet routing is presented and called SmartPacket Routing. The proposed approach provides several opportunities to re-distribute the smartness and decision making among various elements of a network including the packets themselves toward providing a decentralized solution for SDNs. This would bring efficiency and scalability, and therefore also lower environmental footprint for the ever-growing networks. In particular, a region-based representation of the network topology is proposed which is then used to describe the routing actions along …
The high-level contribution of this paper is a simulation-based detailed performance comparison of three different classes of on-demand routing protocols for mobile ad hoc networks: stability-based routing, power-aware routing and load-balanced routing. We choose the Flow-Oriented Routing protocol (FORP), Min-Max Battery Cost Routing (MMBCR) and the traffic interference based Load Balancing Routing (LBR) protocol as representatives of the stability-based routing, poweraware routing and load-balancing routing protocols respectively. FORP incurs the least number of route transitions; while LBR i…
Large scale decentralized communication systems have introduced the new trend towards online routing where routing decisions are performed based on a limited and localized knowledge of the network. Geometrical greedy routing has been among the simplest and most common online routing schemes. A perfect geometrical routing scheme is expected to deliver each packet to the point in the network that is closest to the packet destination. However greedy routing fails to guarantee such delivery as the greedy forwarding decision sometimes leads the packets to localized minimums. This article investigat…
Routing protocols for Mobile Ad Hoc Networks (MANETs) have been extensively studied for more than fifteen years. Position-based routing protocols route packets towards the destination using greedy forwarding (i.e., an intermediate node forwards packets to a neighbor that is closer to the destination than itself). Different position-based protocols use different strategies to pick the neighbor to forward the packet. If a node has no neighbor that is closer to the destination than itself, greedy forwarding fails. In this case, we say there is void (no neighboring nodes) in the direction of the d…
Beihang University（北京航空航天大学）；University of Southampton（南安普顿大学）；Bournemouth University（伯恩茅斯大学）
Data packet routing in aeronautical ad-hoc networks (AANETs) is challenging due to their high-dynamic topology. In this paper, we invoke deep learning (DL) to assist routing in AANETs. We set out from the single objective of minimizing the end-to-end (E2E) delay. Specifically, a deep neural network (DNN) is conceived for mapping the local geographic information observed by the forwarding node into the information required for determining the optimal next hop. The DNN is trained by exploiting the regular mobility pattern of commercial passenger airplanes from historical flight data. After train…
Several emerging classes of applications that run over wireless networks have a need for mathematical models and tools to systematically characterize the reliability of the network. We propose two metrics for measuring the reliability of wireless mesh routing topologies, one for flooding and one for unicast routing. The Flooding Path Probability (FPP) metric measures the end-to-end packet delivery probability when each node broadcasts a packet after hearing from all its upstream neighbors. The Unicast Retransmission Flow (URF) metric measures the end-to-end packet delivery probability when a r…
Wireless network applications, such as, searching, routing, self stabilization and query processing can be modeled as random walks on graphs. Stateless Opportunistic routing technique is a robust distributed routing technique based on random walk approach, where nodes transfer the packets to one of their direct neighbors uniformly, until the packets reach their destinations. Simplicity in execution, fault tolerance, low overhead and robustness to topology changes made it more suitable to wireless sensor networks scenarios. But the main disadvantage of stateless opportunistic routing is estimat…
Routing packets opportunistically is an essential part of multihop ad hoc wireless sensor networks. The existing routing techniques are not adaptive opportunistic. In this paper we have proposed an adaptive opportunistic routing scheme that routes packets opportunistically in order to ensure that packet loss is avoided. Learning and routing are combined in the framework that explores the optimal routing possibilities. In this paper we implemented this Reinforced learning framework using a customer simulator. The experimental results revealed that the scheme is able to exploit the opportunistic…
We give a livelock free routing algorithm for any allowed network. Unlike some other solutions to this problem: 1) packets entering the network have an absolute upper bound on the time to reach their destination; 2) under light loads, packets are delivered to their destinations in nearly optimal time; 3) packets with desired paths far away from congested areas will have routing times far shorter than packets wanting to access congested areas; 4) if the network becomes congested and later clears, the network operates just as it would have when it was initially under a light load. The main ideas…
We consider the resource allocation problem in cooperative wireless networks wherein nodes perform mutual information accumulation. We consider a unicast setting and arbitrary arrival processes at the source node. Source arrivals can be broken down into numerous packets to better exploit the spatial and temporal diversity of the routes available in the network. We devise a linear-program-based algorithm which allocates network resource to meet a certain transmission objective. Given a network, a source with multiple arriving packets and a destination, our algorithm generates a policy that regu…
We propose a universal analysis for static routings on networks and describe the congestion characteristics by the theory. The relation between average transmission time and transmission capacity is described by inequality T0Rc0<=1. For large scale sparse networks, the non-trivial upper bond of transmission capacity Rc0 is limited by Rc0<=1/< 1/k > in some approximate conditions. the theoretical results agree with simulations on BA Networks.
AODV is Ad-hoc On-Demand Distance Vector.A mobile ad-hoc network is a self-configuring network of mobile devices connected by wireless. MANET does not have any fixed infrastructure. The device in a MANET is free to move in any direction and will form the connection as per the requirement of the network. Due to changing topology maintenance of factors like Packet loss, End to End Delay, Number of hops, delivery ratio and controlling the network load is of great challenge. This paper mainly concentrates on reducing the factors such as cost, End-to-End Delay, Network Load and Packet loss in AODV …
Wireless sensor networks sense and monitor real-time events. They supervise a geographic area where a phenomenon is to be monitored. The data in sensor networks have different levels of priority and hence their criticality differs. In order to keep up the real time commitment, the applications need higher transmission rates and reliability in information delivery. In this work we propose a multipath routing algorithm which enables the reliable delivery of data. By controlling the scheduling rate, it is possible to prevent congestion and packet loss in the network. The algorithm provides an eff…
I analyze packet routing on unidirectional ring networks, with an eye towards establishing bounds on the expected length of the queues. Suppose we route packets by a greedy "hot potato" protocol. If packets are inserted by a Bernoulli process and have uniform destinations around the ring, and if the nominal load is kept fixed, then I can construct an upper bound on the expected queue length per node that is independent of the size of the ring. If the packets only travel one or two steps, I can calculate the exact expected queue length for rings of any size. I also show some stability results u…
We consider the impact of scheduling disciplines on performance of routing in the framework of adversarial queuing. We propose an adversarial model which reflects stalling of packets due to transient failures and explicitly incorporates feedback produced by a network when packets are stalled. This adversarial model provides a methodology to study stability of routing protocols when flow-control and congestion-control mechanisms affect the volume of traffic. We show that any scheduling policy that is universally stable, in the regular model of routing that additionally allows packets to have tw…
These Network Coding improves the network operation beyond the traditional routing or store-and-forward, by mixing of data stream within a network. Network coding techniques explicitly minimizes the total no of transmission in wireless network. The Coding-aware routing maximizes the coding opportunity by finding the coding possible path for every packet in the network. Here we propose CORMEN: a new coding-aware routing mechanism based on opportunistic routing. In CORMEN, every node independently can take the decision whether to code packets or not and forwarding of packets is based on the codi…
In this paper, we address the problem of efficient routing in delay tolerant network. We propose a new routing protocol dubbed as ORION. In ORION, only a single copy of a data packet is kept in the network and transmitted, contact by contact, towards the destination. The aim of the ORION routing protocol is twofold: on one hand, it enhances the delivery ratio in networks where an end-to-end path does not necessarily exist, and on the other hand, it minimizes the routing delay and the network overhead to achieve better performance. In ORION, nodes are aware of their neighborhood by the mean of …
We study {\em routing} and {\em scheduling} in packet-switched networks. We assume an adversary that controls the injection time, source, and destination for each packet injected. A set of paths for these packets is {\em admissible} if no link in the network is overloaded. We present the first on-line routing algorithm that finds a set of admissible paths whenever this is feasible. Our algorithm calculates a path for each packet as soon as it is injected at its source using a simple shortest path computation. The length of a link reflects its current congestion. We also show how our algorithm …
Stability is an important issue in order to characterize the performance of a network, and it has become a major topic of study in the last decade. Roughly speaking, a communication network system is said to be stable if the number of packets waiting to be delivered (backlog) is finitely bounded at any one time. In this paper, we introduce a new family of combinatorial structures, which we call universally strong selectors, that are used to provide a set of transmission schedules. Making use of these structures, combined with some known queuing policies, we propose a packet-oblivious routing a…
We present an efficient routing approach for delivering packets in complex networks. On delivering a message from a node to a destination, a node forwards the message to a neighbor by estimating the waiting time along the shortest path from each of its neighbors to the destination. This projected waiting time is dynamical in nature and the path through which a message is delivered would be adapted to the distribution of messages in the network. Implementing the approach on scale-free networks, we show that the present approach performs better than the shortest-path approach and another approac…
Under Water Sensor Networks (UWSN) have different characteristics than terrestrial sensor networks, so routing protocols developed for terrestrial networks are not suitable for UWSN. Underwater routing is comprehended by the underlying network architecture and design of sensor nodes. Therefore, efficient routing protocols need to be developed for UWSNs. We present a Routing scheme based on routing factor (Rf) which is a function of distance and residual energy. We also define Energy Scale Value to rationalize the available energy to take heterogeneity in energies into account. The algorithm ro…
Random 1/2-disk routing in wireless ad-hoc networks is a localized geometric routing scheme in which each node chooses the next relay randomly among the nodes within its transmission range and in the general direction of the destination. We introduce a notion of convergence for geometric routing schemes that not only considers the feasibility of packet delivery through possibly multi-hop relaying, but also requires the packet delivery to occur in a finite number of hops. We derive sufficient conditions that ensure the asymptotic \emph{convergence} of the random 1/2-disk routing scheme based on…
Conventional IP routing protocols are not suitable for multimedia applications which have very stringent Quality-of-Service (QoS) demands and they require a connection oriented service. For multimedia applications it is expected that the router should be able to forward the packet according to the demand of the packet and it is necessary to find a path that satisfies the specific demands of a particular application. In order to address these issues, in this paper, we have presented a QoS aware IP routing protocol where a router stores information about the QoS parameters and routes the packet …
We present and study the Static-Routing-Resiliency problem, motivated by routing on the Internet: Given a graph $G$, a unique destination vertex $d$, and an integer constant $c>0$, does there exist a static and destination-based routing scheme such that the correct delivery of packets from any source $s$ to the destination $d$ is guaranteed so long as (1) no more than $c$ edges fail and (2) there exists a physical path from $s$ to $d$? We embark upon a systematic exploration of this fundamental question in a variety of models (deterministic routing, randomized routing, with packet-duplication,…
Routing in Wireless Sensor Network (WSN) aims to interconnect sensor nodes via single or multi-hop paths. The routes are established to forward data packets from sensor nodes to the sink. Establishing a single path to report each data packet results in increasing energy consumption in WSN, hence, data aggregation routing is used to combine data packets and consequently reduce the number of transmissions. This reduces the routing overhead by eliminating redundant and meaningless data. There are two models for data aggregation routing in WSN: mobile agent and client/server. This paper describes …
Network traffic classification using machine learning techniques has been widely studied. Most existing schemes classify entire traffic flows, but there are major limitations to their practicality. At a network router, the packets need to be processed with minimum delay, so the classifier cannot wait until the end of the flow to make a decision. Furthermore, a complicated machine learning algorithm can be too computationally expensive to implement inside the router. In this paper, we introduce flow-packet hybrid traffic classification (FPHTC), where the router makes a decision per packet based…
Mobile nodes of various routing protocols in Mobile Ad hoc Networks follow different strategies in transmission and receiving of data. Security, packet delivery and routing overhead are important concerns for any protocol during designing them. The presence and absence of malicious nodes in the network affect a lot on the performance of the protocol. This research is mainly focused on the study of the threats, attacks and reasons for malicious behavior of nodes in the network for reactive routing protocols in MANETS. DSR and AODV are the two reactive routing protocols that were considered for …
After a decade of research in low-power data collection, reaching arbitrary nodes has received comparatively little attention. The leading protocol for low-power IPv6 routing, RPL, is no exception, as it is often studied in multipoint-to-point scenarios. As a result, downward routing (from root to node) is still notoriously difficult, holding back the emergence of Internet of Things applications that involve actuation. In this paper, we focus on achieving industrial-grade reliability levels (1e-5 failure rate) in downward routing with RPL. We make every packet count, and classify the different…
Most existing routing strategies to improve transport efficiency have little attention what order should the packets be delivered, just simply used first-in-first-out queue discipline. However, it is far from optimal. In this paper we apply priority queuing discipline to random routing strategy on networks of mobile agents, in which the packets have high priority to transfer directly to their destination despite their order in the queue if their destination are within a communication radius. Numerical experiments show that it not only remarkably improves network throughput and the packet arriv…
A Mobile Ad hoc Network can be said as a collection of mobile nodes, which builds a dynamic topology and a resource constrained network. Due to dynamic nature routing is challenging in MANET. In Mobile Ad hoc Networks different type of attack is possible. By manipulation in routing and attacks against data are reasoned by malicious nodes. MANETs are susceptible to various attacks from selfish nodes. To defeat those problems security is demanded for preventing node from various attacks. Metaheuristic techniques are used for addressing a problem found in ad hoc networks such as routing, security…
The paper presents a quality of service aware routing protocol which provides low latency for high priority packets. Packets are differentiated based on their priority by applying queuing theory. Low priority packets are transferred through less energy paths. The sensor nodes interact with the pivot nodes which in turn communicate with the sink node. This protocol can be applied in monitoring context aware physical environments for critical applications.
MANET is a cooperative network in which nodes are responsible for forwarding as well as routing. Noncooperation is still a big challenge that certainly degrades the performance and reliability of a MANET. This paper presents a novel methodology to overcome routing misbehavior in MANET using Retaliation Model. In this model node misbehavior is watched and an equivalent misbehavior is given in return. This model employs several parameters such as number of packets forwarded, number of packets received for forwarding, packet forwarding ratio etc. to calculate Grade and Bonus Points. The Grade is …
Internet is composed of numbers of independent autonomous systems. BGP is used to disseminate reachability information and establishing path between autonomous systems. Each autonomous system is allowed to select a single route to a destination and then export the selected route to its neighbors. The selection of single best route imposes restrictions on the use of alternative paths during interdomain link failure and thus, incurred packet loss. Packet loss still occurs even when multiple paths exist between source and destination but these paths have not been utilized. To minimize the packet …
In this research paper, an efficient routing protocol for all optical network (AOL) is proposed. The technique uses wavelength division multiplexing (WDM). The proposed one is different from the conventional AOL protocol in transmission of data and control over optical fiber. A set of wavelengths is reserved to transfer control information, which is defined as control wavelengths. Control wavelengths are routed with packet routing scheme and the others are routed with wavelength routing scheme. In connection oriented network only the control packets are sent with the control wavelengths, data …
We consider the problem of routing packets across a multi-hop network consisting of multiple sources of traffic and wireless links while ensuring bounded expected delay. Each packet transmission can be overheard by a random subset of receiver nodes among which the next relay is selected opportunistically. The main challenge in the design of minimum-delay routing policies is balancing the trade-off between routing the packets along the shortest paths to the destination and distributing traffic according to the maximum backpressure. Combining important aspects of shortest path and backpressure r…
In multi-radio wireless mesh networks, one node is eligible to transmit packets over multiple channels to different destination nodes simultaneously. This feature of multi-radio wireless mesh network makes high throughput for the network and increase the chance for multi path routing. This is because the multiple channel availability for transmission decreases the probability of the most elegant problem called as interference problem which is either of interflow and intraflow type. For avoiding the problem like interference and maintaining the constant network performance or increasing the per…
Mobile networks are intriguing in recent years due to their practical implications. Previous routing strategies for improving transport efficiency have little attention what order should the packets be forwarded, just simply used first-in-first-out queue discipline. Here we apply priority queueing discipline to shortest distance routing strategy on mobile networks. Numerical experiments show it not only remarkably improves network throughput and the packet arriving rate, but also reduces average end-to-end delay and the rate of queueing delay. Our work may be helpful in routing strategy design…
VANETs (Vehicular Ad hoc Networks) are highly mobile wireless ad hoc networks and will play an important role in public safety communications and commercial applications. Routing of data in VANETs is a challenging task due to rapidly changing topology and high speed mobility of vehicles. Conventional routing protocols in MANETs (Mobile Ad hoc Networks) are unable to fully address the unique characteristics in vehicular networks. In this paper, we propose EBGR (Edge Node Based Greedy Routing), a reliable greedy position based routing approach to forward packets to the node present in the edge o…
Source-controlled routing has been proposed as a way to improve flexibility of future network architectures, as well as simplifying the data plane. However, if a packet specifies its path, this precludes fast local re-routing within the network. We propose SlickPackets, a novel solution that allows packets to slip around failures by specifying alternate paths in their headers, in the form of compactly-encoded directed acyclic graphs. We show that this can be accomplished with reasonably small packet headers for real network topologies, and results in responsiveness to failures that is competit…
Due to emerging real-time and multimedia applications, efficient routing of information packets in dynamically changing communication network requires that as the load levels, traffic patterns and topology of the network change, the routing policy also adapts. We focused in this paper on QoS based routing by developing a neuro-dynamic programming to construct dynamic state dependent routing policies. We propose an approach based on adaptive algorithm for packet routing using reinforcement learning which optimizes two criteria: cumulative cost path and end-to-end delay. Numerical results obtain…
We suggest and analyze algorithms for routing in multi-hop wireless ad-hoc networks that exploit mutual information accumulation as the physical layer transmission scheme, and are capable of routing multiple packet streams (commodities) when only the average channel state information is present and that only locally. The proposed algorithms are modifications of the Diversity Backpressure (DIVBAR) algorithm, under which the packet whose commodity has the largest "backpressure metric" is chosen to be transmitted and is forwarded through the link with the largest differential backlog (queue lengt…
