360$^o$ video is an essential component of VR/AR/MR systems that provides immersive experience to the users. However, 360$^o$ video is associated with high bandwidth requirements. The required bandwidth can be reduced by exploiting the fact that users are interested in viewing only a part of the video scene and that users request viewports that overlap with each other. Motivated by the findings of recent works where the benefits of caching video tiles at edge servers instead of caching entire 360$^o$ videos were shown, in this paper, we introduce the concept of virtual viewports that have the same number of tiles with the original viewports. The tiles forming these viewports are the most popular ones for each video and are determined by the users' requests. Then, we propose a proactive caching scheme that assumes unknown videos' and viewports' popularity. Our scheme determines which videos to cache as well as which is the optimal virtual viewport per video. Virtual viewports permit to lower the dimensionality of the cache optimization problem. To solve the problem, we first formulate the content placement of 360$^o$ videos in edge cache networks as a Markov Decision Process (MDP), and then we determine the optimal caching placement using the Deep Q-Network (DQN) algorithm. The proposed solution aims at maximizing the overall quality of the 360$^o$ videos delivered to the end-users by caching the most popular 360$^o$ videos at base quality along with a virtual viewport in high quality. We extensively evaluate the performance of the proposed system and compare it with that of known systems such as LFU, LRU, FIFO, over both synthetic and real 360$^o$ video traces. The results reveal the large benefits coming from proactive caching of virtual viewports instead of the original ones in terms of the overall quality of the rendered viewports, the cache hit ratio, and the servicing cost.
University of Minnesota（明尼苏达大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
With the tremendous growth of data traffic over wired and wireless networks along with the increasing number of rich-media applications, caching is envisioned to play a critical role in next-generation networks. To intelligently prefetch and store contents, a cache node should be able to learn what and when to cache. Considering the geographical and temporal content popularity dynamics, the limited available storage at cache nodes, as well as the interactive in uence of caching decisions in networked caching settings, developing effective caching policies is practically challenging. In respons…
Caching is envisioned to play a critical role in next-generation content delivery infrastructure, cellular networks, and Internet architectures. By smartly storing the most popular contents at the storage-enabled network entities during off-peak demand instances, caching can benefit both network infrastructure as well as end users, during on-peak periods. In this context, distributing the limited storage capacity across network entities calls for decentralized caching schemes. Many practical caching systems involve a parent caching node connected to multiple leaf nodes to serve user file reque…
The growing demand on high-quality and low-latency multimedia services has led to much interest in edge caching techniques. Motivated by this, we in this paper consider edge caching at the base stations with unknown content popularity distributions. To solve the dynamic control problem of making caching decisions, we propose a deep actor-critic reinforcement learning based multi-agent framework with the aim to minimize the overall average transmission delay. To evaluate the proposed framework, we compare the learning-based performance with three other caching policies, namely least recently us…
IRIDA Research Centre for Communication Technologies（IRIDA通信技术研究中心）；University of Cyprus（塞浦路斯大学）；Ionian University（爱奥尼亚大学）；Aalto University（阿尔托大学）；Rutgers University（罗格斯大学）
Mobile networks are experiencing tremendous increase in data volume and user density. An efficient technique to alleviate this issue is to bring the data closer to the users by exploiting the caches of edge network nodes, such as fixed or mobile access points and even user devices. Meanwhile, the fusion of machine learning and wireless networks offers a viable way for network optimization as opposed to traditional optimization approaches which incur high complexity, or fail to provide optimal solutions. Among the various machine learning categories, reinforcement learning operates in an online…
Southeast University（东南大学）；Purple Mountain Laboratories（紫金山实验室）；Georgia Institute of Technology（佐治亚理工学院）
For ultra-dense networks with wireless backhaul, caching strategy at small base stations (SBSs), usually with limited storage, is critical to meet massive high data rate requests. Since the content popularity profile varies with time in an unknown way, we exploit reinforcement learning (RL) to design a cooperative caching strategy with maximum-distance separable (MDS) coding. We model the MDS coding based cooperative caching as a Markov decision process to capture the popularity dynamics and maximize the long-term expected cumulative traffic load served directly by the SBSs without accessing t…
The Internet of Things (IoT) has been continuously rising in the past few years, and its potentials are now more apparent. However, transient data generation and limited energy resources are the major bottlenecks of these networks. Besides, minimum delay and other conventional quality of service measurements are still valid requirements to meet. An efficient caching policy can help meet the standard quality of service requirements while bypassing IoT networks' specific limitations. Adopting deep reinforcement learning (DRL) algorithms enables us to develop an effective caching scheme without t…
Intel Corporation（英特尔公司）；University of Southern California（南加州大学）
We consider a system comprising a file library and a network with a server and multiple users equipped with cache memories. The system operates in two phases: a prefetching phase, where users load their caches with parts of contents from the library, and a delivery phase, where users request files from the library and the server needs to send the uncached parts of the requested files to the users. For the case where the users' caches are arbitrarily loaded, we propose an algorithm based on deep reinforcement learning to minimize the delay of delivering requested contents to the users in the de…
University of Electronic Science and Technology of China（电子科技大学）；Simula Metropolitan Center for Digital Engineering（Simula都市数字工程中心）；University of Oslo（奥斯陆大学）
Vehicular Edge Computing (VEC) is a promising paradigm to enable huge amount of data and multimedia content to be cached in proximity to vehicles. However, high mobility of vehicles and dynamic wireless channel condition make it challenge to design an optimal content caching policy. Further, with much sensitive personal information, vehicles may be not willing to caching their contents to an untrusted caching provider. Deep Reinforcement Learning (DRL) is an emerging technique to solve the problem with high-dimensional and time-varying features. Permission blockchain is able to establish a sec…
Southeast University（东南大学）；Xidian University（西安电子科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Oulu（奥卢大学）；University of Manchester（曼彻斯特大学）；Harbin Institute of Technology（哈尔滨工业大学）
In this paper, the distributed edge caching problem in fog radio access networks (F-RANs) is investigated. By considering the unknown spatio-temporal content popularity and user preference, a user request model based on hidden Markov process is proposed to characterize the fluctuant spatio-temporal traffic demands in F-RANs. Then, the Q-learning method based on the reinforcement learning (RL) framework is put forth to seek the optimal caching policy in a distributed manner, which enables fog access points (F-APs) to learn and track the potential dynamic process without extra communications cos…
The demand and usage of 360$^{\circ}$ video services are expected to increase. However, despite these services being highly bandwidth intensive, not much is known about the potential value that basic bandwidth saving techniques such as server or edge-network on-demand caching (e.g., in a CDN) could have when used for delivery of such services. This problem is both important and complicated as client-side solutions have been developed that split the full 360$^{\circ}$ view into multiple tiles, and adapt the quality of the downloaded tiles based on the user's expected viewing direction and bandw…
Southeast University（东南大学）；Purple Mountain Laboratories（紫金山实验室）；Singapore University of Technology and Design（新加坡科技设计大学）
Mobile edge computing (MEC) is a prominent computing paradigm which expands the application fields of wireless communication. Due to the limitation of the capacities of user equipments and MEC servers, edge caching (EC) optimization is crucial to the effective utilization of the caching resources in MEC-enabled wireless networks. However, the dynamics and complexities of content popularities over space and time as well as the privacy preservation of users pose significant challenges to EC optimization. In this paper, a privacy-preserving distributed deep deterministic policy gradient (P2D3PG) …
Performance metrics-driven context caching has a profound impact on throughput and response time in distributed context management systems for real-time context queries. This paper proposes a reinforcement learning based approach to adaptively cache context with the objective of minimizing the cost incurred by context management systems in responding to context queries. Our novel algorithms enable context queries and sub-queries to reuse and repurpose cached context in an efficient manner. This approach is distinctive to traditional data caching approaches by three main features. First, we mak…
University of Minnesota（明尼苏达大学）；King Juan Carlos University（胡安·卡洛斯国王大学）
Small base stations (SBs) of fifth-generation (5G) cellular networks are envisioned to have storage devices to locally serve requests for reusable and popular contents by \emph{caching} them at the edge of the network, close to the end users. The ultimate goal is to shift part of the predictable load on the back-haul links, from on-peak to off-peak periods, contributing to a better overall network performance and service experience. To enable the SBs with efficient \textit{fetch-cache} decision-making schemes operating in dynamic settings, this paper introduces simple but flexible generic time…
With data durability, high access speed, low power efficiency and byte addressability, NVMe and SSD, which are acknowledged representatives of emerging storage technologies, have been applied broadly in many areas. However, one key issue with high-performance adoption of these technologies is how to properly define intelligent cache layers such that the performance gap between emerging technologies and main memory can be well bridged. To this end, we propose Phoebe, a reuse-aware reinforcement learning framework for the optimal online caching that is applicable for a wide range of emerging sto…
Jiangnan University（江南大学）；State Key Laboratory of Integrated Services Networks (Xidian University)（综合业务网理论及关键技术国家重点实验室（西安电子科技大学））；Qualcomm（高通公司）；Tsinghua University（清华大学）；University of Kent（肯特大学）；Banma Network Technology Co., Ltd.（斑马网络技术有限公司）
The vehicular edge computing (VEC) can cache contents in different RSUs at the network edge to support the real-time vehicular applications. In VEC, owing to the high-mobility characteristics of vehicles, it is necessary to cache the user data in advance and learn the most popular and interesting contents for vehicular users. Since user data usually contains privacy information, users are reluctant to share their data with others. To solve this problem, traditional federated learning (FL) needs to update the global model synchronously through aggregating all users' local models to protect user…
Cooperative Edge Caching Based on Elastic Federated and Multi-Agent Deep Reinforcement Learning in Next-Generation Network
School of Internet of Things Engineering, Jiangnan University（江南大学物联网工程学院）；State Key Laboratory of Integrated Services Networks (Xidian University)（西安电子科技大学综合业务网理论及关键技术国家重点实验室）；Qualcomm（高通公司）；School of Engineering, University of Kent（肯特大学工程学院）；School of Engineering, Hong Kong University of Science and Technology（香港科技大学工程学院）
Tandon School of Engineering, New York University（纽约大学坦登工程学院）；New York Institute of Technology（纽约理工学院）
Edge caching will play a critical role in facilitating the emerging content-rich applications. However, it faces many new challenges, in particular, the highly dynamic content popularity and the heterogeneous caching configurations. In this paper, we propose Cocktail Edge Caching, that tackles the dynamic popularity and heterogeneity through ensemble learning. Instead of trying to find a single dominating caching policy for all the caching scenarios, we employ an ensemble of constituent caching policies and adaptively select the best-performing policy to control the cache. Towards this goal, w…
University of Perugia（佩鲁贾大学）；INFN - Sezione di Perugia（意大利国家核物理研究所佩鲁贾分部）；University "G. D’Annunzio" of Chieti-Pescara（基耶蒂-佩斯卡拉“加布里埃尔·邓南遮”大学）
The continuous growth of data production in almost all scientific areas raises new problems in data access and management, especially in a scenario where the end-users, as well as the resources that they can access, are worldwide distributed. This work is focused on the data caching management in a Data Lake infrastructure in the context of the High Energy Physics field. We are proposing an autonomous method, based on Reinforcement Learning techniques, to improve the user experience and to contain the maintenance costs of the infrastructure.
Singapore University of Technology and Design（新加坡科技设计大学）；Villanova University（维拉诺瓦大学）；Pusan National University（釜山大学）；Princeton University（普林斯顿大学）；Kyung Hee University（庆熙大学）
This work exploits the advantages of two prominent techniques in future communication networks, namely caching and non-orthogonal multiple access (NOMA). Particularly, a system with Rayleigh fading channels and cache-enabled users is analyzed. It is shown that the caching-NOMA combination provides a new opportunity of cache hit which enhances the cache utility as well as the effectiveness of NOMA. Importantly, this comes without requiring users' collaboration, and thus, avoids many complicated issues such as users' privacy and security, selfishness, etc. In order to optimize users' quality of …
Multi-Time Scale Service Caching and Pricing in MEC Systems with Dynamic Program Popularity
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；School of Cyber Science and Engineering, Xi’an Jiaotong University（西安交通大学网络空间安全学院）
Attention-Enhanced Prioritized Proximal Policy Optimization for Adaptive Edge Caching
Lassonde School of Engineering at York University（约克大学拉松德工程学院）；School of Electronic and Computer Engineering at the Hong Kong University of Science and Technology（香港科技大学电子与计算机工程学院）；School of Engineering and Applied Science at the University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）；School of Electrical and Computer Engineering at the University of Tehran（德黑兰大学电气与计算机工程学院）
Emergency Computing: An Adaptive Collaborative Inference Method Based on Hierarchical Reinforcement Learning
School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications（北京邮电大学计算机学院（国家示范性软件学院））；School of Electronic Engineering, Beijing University of Posts and Telecommunications（北京邮电大学电子工程学院）
Binghamton University, State University of New York（纽约州立大学宾汉姆顿分校）；Indian Institute of Science（印度科学学院）
We consider the problem of content caching at the wireless edge to serve a set of end users via unreliable wireless channels so as to minimize the average latency experienced by end users due to the constrained wireless edge cache capacity. We formulate this problem as a Markov decision process, or more specifically a restless multi-armed bandit problem, which is provably hard to solve. We begin by investigating a discounted counterpart, and prove that it admits an optimal policy of the threshold-type. We then show that this result also holds for average latency problem. Using this structural …
TDC-Cache: A Trustworthy Decentralized Cooperative Caching Framework for Web3.0
School of Electronic and Optical Engineering, Nanjing University of Science and Technology（南京理工大学电子与光学工程学院）；College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；National Mobile Communications Research Laboratory, Southeast University（东南大学国家移动通信研究中心）；Purple Mountain Laboratories（紫金山实验室）；School of Electrical Engineering and Telecommunications, The University of New South Wales（新南威尔士大学电子与电信工程学院）
MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation
School of Computer Science, Peking University, Beijing, China（北京大学计算机科学学院，北京，中国）；School of Information, Renmin University of China, Beijing, China（中国人民大学信息学院，北京，中国）
Khoury College of Computer Sciences, Northeastern University（东北大学库里计算机科学学院）
Return caching is a recent strategy that enables efficient minibatch training with multistep estimators (e.g. the λ-return) for deep reinforcement learning. By precomputing return estimates in sequential batches and then storing the results in an auxiliary data structure for later sampling, the average computation spent per estimate can be greatly reduced. Still, the efficiency of return caching could be improved, particularly with regard to its large memory usage and repetitive data copies. We propose a new data structure, the Virtual Replay Cache (VRC), to address these shortcomings. When le…
Recently, we have struck the balance between the information freshness, in terms of age of information (AoI), experienced by users and energy consumed by sensors, by appropriately activating sensors to update their current status in caching enabled Internet of Things (IoT) networks [1]. To solve this problem, we cast the corresponding status update procedure as a continuing Markov Decision Process (MDP) (i.e., without termination states), where the number of state-action pairs increases exponentially with respect to the number of considered sensors and users. Moreover, to circumvent the curse …
Two-Timescale Model Caching and Resource Allocation for Edge-Enabled AI-Generated Content Services
Xiamen University（厦门大学）；University of Hong Kong（香港大学）；Tsinghua University（清华大学）；University at Buffalo–SUNY（纽约州立大学布法罗分校）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；York University（约克大学）；University of Electronic Science and Technology of China（电子科技大学）；Sungkyunkwan University（成均馆大学）
This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and a…
Recent approaches for learning policies to improve caching, target just one out of the prefetching, admission and eviction processes. In contrast, we propose an end to end pipeline to learn all three policies using machine learning. We also take inspiration from the success of pretraining on large corpora to learn specialized embeddings for the task. We model prefetching as a sequence prediction task based on past misses. Following previous works suggesting that frequency and recency are the two orthogonal fundamental attributes for caching, we use an online reinforcement learning technique to…
University of Cambridge（剑桥大学）；University of Hamburg（汉堡大学）
Learning effective configurations in computer systems without hand-crafting models for every parameter is a long-standing problem. This paper investigates the use of deep reinforcement learning for runtime parameters of cloud databases under latency constraints. Cloud services serve up to thousands of concurrent requests per second and can adjust critical parameters by leveraging performance metrics. In this work, we use continuous deep reinforcement learning to learn optimal cache expirations for HTTP caching in content delivery networks. To this end, we introduce a technique for asynchronous…
Girton College（格顿学院）；University of Cambridge（剑桥大学）
This study investigates the use of reinforcement learning to guide a general purpose cache manager decisions. Cache managers directly impact the overall performance of computer systems. They govern decisions about which objects should be cached, the duration they should be cached for, and decides on which objects to evict from the cache if it is full. These three decisions impact both the cache hit rate and size of the storage that is needed to achieve that cache hit rate. An optimal cache manager will avoid unnecessary operations, maximise the cache hit rate which results in fewer round trips…
Reward Augmentation in Reinforcement Learning for Testing Distributed Systems
Max Planck Institute for Software Systems (MPI-SWS)（马克斯·普朗克软件系统研究所）；CNRS（法国国家科学研究中心）；Ecole Polytechnique（巴黎综合理工学院）；Chennai Mathematical Institute（金奈数学研究所）；IRIF（法国信息学研究所）；Universite Paris Cite（巴黎西岱大学）
Refining Hybrid Genetic Search for CVRP via Reinforcement Learning-Finetuned LLM
School of Teacher Education, Nanjing University of Information Science and Technology, China（南京信息工程大学教师教育学院）；Nanyang Technological University, Singapore（新加坡南洋理工大学）；Singapore Management University, Singapore（新加坡管理大学）
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；PHENIKAA University（phenikaa大学）；Nanyang Technological University（南洋理工大学）
Future Internet involves several emerging technologies such as 5G and beyond 5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and Internet of Things (IoTs). Moreover, future Internet becomes heterogeneous and decentralized with a large number of involved network entities. Each entity may need to make its local decision to improve the network performance under dynamic and uncertain network environments. Standard learning algorithms such as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) have been recently used to enable each network entity a…
Cache-Efficient Posterior Sampling for Reinforcement Learning with LLM-Derived Priors Across Discrete and Continuous Domains
Department of Computer Science, Iowa State University, Ames, IA, USA（计算机科学系，爱荷华州立大学）；Department of Civil, Construction and Environmental Engineering, Iowa State University, Ames, IA, USA（土木、建设与环境工程系，爱荷华州立大学）
ProCAVE: A Self-Adaptive, Full-Lifecycle Edge Caching Framework for Video Streaming via Predictive Bandwidth Estimation and Preference-Aware Deep Reinforcement Learning
Tarbiat Modares University（塔比尔·莫达雷斯大学）；ICT Research Institute（信息通信技术研究所）
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学网络安全学院）
Caching and rate allocation are two promising approaches to support video streaming over wireless network. However, existing rate allocation designs do not fully exploit the advantages of the two approaches. This paper investigates the problem of cache-enabled QoE-driven video rate allocation problem. We establish a mathematical model for this problem, and point out that it is difficult to solve the problem with traditional dynamic programming. Then we propose a deep reinforcement learning approaches to solve it. First, we model the problem as a Markov decision problem. Then we present a deep …
