In recent years several architectures have been proposed to learn embodied agents complex self-awareness models. In this paper, dynamic incremental self-awareness (SA) models are proposed that allow experiences done by an agent to be modeled in a hierarchical fashion, starting from more simple situations to more structured ones. Each situation is learned from subsets of private agent perception data as a model capable to predict normal behaviors and detect abnormalities. Hierarchical SA models have been already proposed using low dimensional sensorial inputs. In this work, a hierarchical model is introduced by means of a cross-modal Generative Adversarial Networks (GANs) processing high dimensional visual data. Different levels of the GANs are detected in a self-supervised manner using GANs discriminators decision boundaries. Real experiments on semi-autonomous ground vehicles are presented.
Understanding the mechanism of generative adversarial networks (GANs) helps us better use GANs for downstream applications. Existing efforts mainly target interpreting unconditional models, leaving it less explored how a conditional GAN learns to render images regarding various categories. This work fills in this gap by investigating how a class conditional generator unifies the synthesis of multiple classes. For this purpose, we dive into the widely used class-conditional batch normalization (CCBN), and observe that each feature channel is activated at varying degrees given different categori…
The Chinese University of Hong Kong（香港中文大学）；The Australian National University（澳大利亚国立大学）；University of California, Los Angeles（加州大学洛杉矶分校）
The success of Generative Adversarial Networks (GANs) is largely built upon the adversarial training between a generator (G) and a discriminator (D). They are expected to reach a certain equilibrium where D cannot distinguish the generated images from the real ones. However, such an equilibrium is rarely achieved in practical GAN training, instead, D almost always surpasses G. We attribute one of its sources to the information asymmetry between D and G. We observe that D learns its own visual attention when determining whether an image is real or fake, but G has no explicit clue on which regio…
University of Genoa（热那亚大学）；Queen Mary University of London（伦敦玛丽女王大学）；Carlos III University of Madrid（马德里卡洛斯三世大学）
This paper focuses on multi-sensor anomaly detection for moving cognitive agents using both external and private first-person visual observations. Both observation types are used to characterize agents' motion in a given environment. The proposed method generates locally uniform motion models by dividing a Gaussian process that approximates agents' displacements on the scene and provides a Shared Level (SL) self-awareness based on Environment Centered (EC) models. Such models are then used to train in a semi-unsupervised way a set of Generative Adversarial Networks (GANs) that produce an estim…
Training GANs under limited data often leads to discriminator overfitting and memorization issues, causing divergent training. Existing approaches mitigate the overfitting by employing data augmentations, model regularization, or attention mechanisms. However, they ignore the frequency bias of GANs and take poor consideration towards frequency information, especially high-frequency signals that contain rich details. To fully utilize the frequency information of limited data, this paper proposes FreGAN, which raises the model's frequency awareness and draws more attention to producing high-freq…
Generative adversarial networks (GANs), e.g., StyleGAN2, play a vital role in various image generation and synthesis tasks, yet their notoriously high computational cost hinders their efficient deployment on edge devices. Directly applying generic compression approaches yields poor results on GANs, which motivates a number of recent GAN compression works. While prior works mainly accelerate conditional GANs, e.g., pix2pix and CycleGAN, compressing state-of-the-art unconditional GANs has rarely been explored and is more challenging. In this paper, we propose novel approaches for unconditional G…
Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Maryland College Park（马里兰大学帕克分校）
本文提出 Hijack-GAN，通过迭代方案在仅具黑盒访问权限时控制预训练 GAN 的图像生成，揭示公开发布模型可能被非预期重用的安全风险。
We propose embodied scene-aware human pose estimation where we estimate 3D poses based on a simulated agent's proprioception and scene awareness, along with external third-person observations. Unlike prior methods that often resort to multistage optimization, non-causal inference, and complex contact modeling to estimate human pose and human scene interactions, our method is one-stage, causal, and recovers global 3D human poses in a simulated environment. Since 2D third-person observations are coupled with the camera pose, we propose to disentangle the camera pose and use a multi-step projecti…
CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Harbin Institute of Technology（哈尔滨工业大学）；University of Science and Technology of China（中国科学技术大学）；CUHK-Shenzhen（香港中文大学（深圳））；Fudan University（复旦大学）；Dalian University of Technology（大连理工大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Oxford（牛津大学）
2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness
School of Computer Science, Peking University（北京大学计算机科学学院）；ZTE Corporation（中兴通讯）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Computer Science, China University of Geosciences (Wuhan)（中国地质大学（武汉）计算机科学学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）
