Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
City University of Hong Kong；University of Illinois Urbana-Champaign；Tencent；Institute of Artificial Intelligence, China Telecom (TeleAI)；Tsinghua University. Corresponding Author
跨领域离线强化学习旨在利用有限的目标域数据集和可能具有充分数据覆盖的源域数据集来训练高性能的智能体。由于源域和目标域底层动力学不一致，简单合并两个数据集可能导致性能下降。近期研究通过选择动力学对齐良好的源域样本来解决此问题，但本文指出仅靠动力学对齐是不够的，通过分析现有框架的局限性并推导出新的目标域子最优界，证明了价值对齐的必要性。本文提出Dynamics- and Value-aligned Data Filtering (DVDF) 方法，一种新的统一跨领域强化学习框架，选择在动力学和价值上均表现出强对齐的源域样本。实验研究了多种动力学转移场景，包括运动学和形态学转移，并在各种任务和数据集上评估DVDF，即使在目标域数据极有限的挑战性设置中也表现良好。广泛实验表明，DVDF在强基线方法上表现优异，具有显著改进。
英文摘要：
Cross-domain offline reinforcement learning (RL) aims to train a well-performing agent in the target environment, leveraging both a limited target domain dataset and a source domain dataset with (possibly) sufficient data coverage. Due to the underlying dynamics misalignment between source and target domains, naively merging the two datasets may incur inferior performance. Recent advances address this issue by selectively leveraging source domain samples whose dynamics align well with the target domain. However, our work demonstrates that dynamics alignment alone is insufficient, by examining the limitations of prior frameworks and deriving a new target domain sub-optimality bound for the policy learned on the source domain. More importantly, our theory underscores an additional need for \textit{value alignment}, i.e., selecting high-quality, high-value samples from the source domain, a critical dimension overlooked by existing works. Motivated by such theoretical insight, we propose \textbf{\underline{D}}ynamics- and \textbf{\underline{V}}alue-aligned \textbf{\underline{D}}ata \textbf{\underline{F}}iltering (DVDF) method, a novel unified cross-domain RL framework that selectively incorporates source domain samples exhibiting strong alignment in \textit{both dynamics and values}. We empirically study a range of dynamics shift scenarios, including kinematic and morphology shifts, and evaluate DVDF on various tasks and datasets, even in the challenging setting where the target domain dataset contains an extremely limited amount of data. Extensive experiments demonstrate that DVDF consistently outperforms strong baselines with significant improvements.
Sample-Efficient Policy Constraint Offline Deep Reinforcement Learning based on Sample Filtering
Guangdong Key Laboratory of Intelligent Morphing Mechanisms（广东智能变形机制与自适应机器人重点实验室）；the Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳研究院）；Faculty of Robot Science and Engineering（机器人科学与工程学院）；Northeastern University（东北大学）；School of Systems Engineering（系统工程学院）；City University of Hong Kong（香港城市大学）
Active Advantage-Aligned Online Reinforcement Learning with Offline Data
Department of Computer Science, University of Chicago（计算机科学系，芝加哥大学）；Department of Statistics and Data Science, Yale University（统计与数据科学系，耶鲁大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets
City University of Hong Kong；Tencent；Institute of Artificial Intelligence (TeleAI), China Telecom；Institute of Automation, Chinese Academy of Sciences. Corresponding Author
Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络科学与技术学院）；MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University（教育部人工智能重点实验室）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Computer Science and Engineering, Lehigh University（利哈伊大学计算机科学与工程学院）
Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
Northwestern Polytechnical University（西北工业大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）；Fudan University（复旦大学）；Harbin Institute of Technology（哈尔滨工业大学）；Tsinghua University（清华大学）；The Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
L3S Research Center（L3S研究中心）；Leibniz University Hannover（汉诺威莱布尼茨大学）
The success of deep reinforcement learning (DRL) hinges on the availability of training data, which is typically obtained via a large number of environment interactions. In many real-world scenarios, costs and risks are associated with gathering these data. The field of offline reinforcement learning addresses these issues through outsourcing the collection of data to a domain expert or a carefully monitored program and subsequently searching for a batch-constrained optimal policy. With the emergence of data markets, an alternative to constructing a dataset in-house is to purchase external dat…
Offline policy optimization could have a large impact on many real-world decision-making problems, as online learning may be infeasible in many applications. Importance sampling and its variants are a commonly used type of estimator in offline policy evaluation, and such estimators typically do not require assumptions on the properties and representational capabilities of value function or decision process model function classes. In this paper, we identify an important overfitting phenomenon in optimizing the importance weighted return, in which it may be possible for the learned policy to ess…
Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
Georgia Institute of Technology（佐治亚理工学院）；Amazon Alexa AI（亚马逊Alexa人工智能）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Southern California（南加州大学）
Multi-armed bandits (MAB) provide a principled online learning approach to attain the balance between exploration and exploitation. Due to the superior performance and low feedback learning without the learning to act in multiple situations, Multi-armed Bandits drawing widespread attention in applications ranging such as recommender systems. Likewise, within the recommender system, collaborative filtering (CF) is arguably the earliest and most influential method in the recommender system. Crucially, new users and an ever-changing pool of recommended items are the challenges that recommender sy…
Viadeo；Université Paris 1 Panthéon - Sorbonne（巴黎第一大学先贤祠-索邦大学）
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical data (via offline evaluation). This paper presents a new application of a weighted offline evaluation to reduce this bias for collaborative filtering algorithms.
We introduce quantile filtered imitation learning (QFIL), a novel policy improvement operator designed for offline reinforcement learning. QFIL performs policy improvement by running imitation learning on a filtered version of the offline dataset. The filtering process removes $ s,a $ pairs whose estimated Q values fall below a given quantile of the pushforward distribution over values induced by sampling actions from the behavior policy. The definitions of both the pushforward Q distribution and resulting value function quantile are key contributions of our method. We prove that QFIL gives us…
Streaming Adversarial Robustness in Fuzzy ARTMAP: Mechanism-Aligned Evaluation, Progressive Training, and Interpretable Diagnostics
Department of Electrical and Computer Engineering（电气与计算机工程系）；Kummer Institute Center for Artificial Intelligence and Autonomous Systems (KICAIAS)（人工智能与自主系统中心（KICAIAS））；Missouri University of Science and Technology（密苏里科技大学）
Data processing tasks over graphs couple the data residing over the nodes with the topology through graph signal processing tools. Graph filters are one such prominent tool, having been used in applications such as denoising, interpolation, and classification. However, they are mainly used on fixed graphs although many networks grow in practice, with nodes continually attaching to the topology. Re-training the filter every time a new node attaches is computationally demanding; hence an online learning solution that adapts to the evolving graph is needed. We propose an online update of the filt…
Viadeo；Université Paris 1 Panthéon – Sorbonne（巴黎第一大学先贤祠-索邦大学）
Recommendation systems have been integrated into the majority of large online systems. They tailor those systems to individual users by filtering and ranking information according to user profiles. This adaptation process influences the way users interact with the system and, as a consequence, increases the difficulty of evaluating a recommendation algorithm with historical data (via offline evaluation). This paper analyses this evaluation bias and proposes a simple item weighting solution that reduces its impact. The efficiency of the proposed solution is evaluated on real world data extracte…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Artificial Intelligence Research Institute（人工智能研究院）；University of California, Irvine（加州大学欧文分校）
Offline Reinforcement Learning (RL) addresses the problem of sequential decision-making by learning optimal policy through pre-collected data, without interacting with the environment. As yet, it has remained somewhat impractical, because one rarely knows the reward explicitly and it is hard to distill it retrospectively. Here, we show that an imitating agent can still learn the desired behavior merely from observing the expert, despite the absence of explicit rewards or action labels. In our method, AILOT (Aligned Imitation Learning via Optimal Transport), we involve special representation of…
DDoS attacks are simple, effective, and still pose a significant threat even after more than two decades. Given the recent success in machine learning, it is interesting to investigate how we can leverage deep learning to filter out application layer attack requests. There are challenges in adopting deep learning solutions due to the ever-changing profiles, the lack of labeled data, and constraints in the online setting. Offline unsupervised learning methods can sidestep these hurdles by learning an anomaly detector $N$ from the normal-day traffic ${\mathcal N}$. However, anomaly detection doe…
Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
University of Cambridge（剑桥大学）；OATML, University of Oxford（牛津大学OATML实验室）；University of Toronto（多伦多大学）；Shanghai Jiao Tong University（上海交通大学）；Samsung AI Center（三星人工智能中心）
University of Klagenfurt（克拉根福大学）；Silicon Austria Labs GmbH（Silicon Austria Labs 有限公司）
This article proposes and evaluates a technique to predict the level of interference in wireless networks. We design a recursive predictor that estimates future interference values by filtering measured interference at a given location. The predictor's parameterization is done offline by translating the autocorrelation of interference into an autoregressive moving average (ARMA) representation. This ARMA model is inserted into a steady-state Kalman filter enabling nodes to predict with low computational effort. Results show a good accuracy of predicted values versus true values for relevant ti…
Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
AIRI, Skoltech Moscow Russian Federation（AIRI，斯克里普钦斯基因工学院莫斯科俄罗斯联邦）；Sber AI Lab, Skoltech Moscow Russian Federation（Sber AI Lab，斯克里普钦斯基因工学院莫斯科俄罗斯联邦）；Sber AI Lab Moscow Russian Federation（Sber AI Lab，莫斯科斯克里普钦斯基因工学院俄罗斯联邦）；Sber AI Lab, HSE University Moscow Russian Federation（Sber AI Lab，莫斯科高等经济大学俄罗斯联邦）
HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学 HiTZ 中心）；Facebook AI Research（Facebook 人工智能研究院）
Recent research on cross-lingual word embeddings has been dominated by unsupervised mapping approaches that align monolingual embeddings. Such methods critically rely on those embeddings having a similar structure, but it was recently shown that the separate training in different languages causes departures from this assumption. In this paper, we propose an alternative approach that does not have this limitation, while requiring a weak seed dictionary (e.g., a list of identical words) as the only form of supervision. Rather than aligning two fixed embedding spaces, our method works by fixing t…
Chinese University of Hongkong (Shenzhen)（香港中文大学（深圳））；Chapman University（查普曼大学）；Boston University（波士顿大学）
We propose a novel supervised learning approach for political ideology prediction (PIP) that is capable of predicting out-of-distribution inputs. This problem is motivated by the fact that manual data-labeling is expensive, while self-reported labels are often scarce and exhibit significant selection bias. We propose a novel statistical model that decomposes the document embeddings into a linear superposition of two vectors; a latent neutral \emph{context} vector independent of ideology, and a latent \emph{position} vector aligned with ideology. We train an end-to-end model that has intermedia…
