Generating higher-resolution human-centric scenes with details and controls remains a challenge for existing text-to-image diffusion models. This challenge stems from limited training image size, text encoder capacity (limited tokens), and the inherent difficulty of generating complex scenes involving multiple humans. While current methods attempted to address training size limit only, they often yielded human-centric scenes with severe artifacts. We propose BeyondScene, a novel framework that overcomes prior limitations, generating exquisite higher-resolution (over 8K) human-centric scenes with exceptional text-image correspondence and naturalness using existing pretrained diffusion models. BeyondScene employs a staged and hierarchical approach to initially generate a detailed base image focusing on crucial elements in instance creation for multiple humans and detailed descriptions beyond token limit of diffusion model, and then to seamlessly convert the base image to a higher-resolution output, exceeding training image size and incorporating details aware of text and instances via our novel instance-aware hierarchical enlargement process that consists of our proposed high-frequency injected forward diffusion and adaptive joint diffusion. BeyondScene surpasses existing methods in terms of correspondence with detailed text descriptions and naturalness, paving the way for advanced applications in higher-resolution human-centric scene creation beyond the capacity of pretrained diffusion models without costly retraining. Project page: https://janeyeon.github.io/beyond-scene.
DreamFit: Garment-Centric Human Generation via a Lightweight Anything-Dressing Encoder
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；ByteDance（字节跳动）；Sun Yat-sen University（中山大学）
Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Tasks
University of Kaiserslautern-Landau（凯泽斯劳滕-兰道大学）；Deutsches Forschungszentrum für Künstliche Intelligenz（德国人工智能研究中心）；ETH Zürich（苏黎世联邦理工学院）；Technical University of Munich（慕尼黑工业大学）
Concept-Centric Token Interpretation for Vector-Quantized Generative Models
School of Computing, University of Georgia（佐治亚大学计算学院）；Department of Data Science, New Jersey Institute of Technology（新泽西理工学院数据科学系）；Department of Computer Science, New York University（纽约大学计算机科学系）
Meta AI Research（Meta人工智能研究院）；Inria（法国国家信息与自动化研究所）
Discriminative self-supervised learning allows training models on any random group of internet images, and possibly recover salient information that helps differentiate between the images. Applied to ImageNet, this leads to object centric features that perform on par with supervised features on most object-centric downstream tasks. In this work, we question if using this ability, we can learn any salient and more representative information present in diverse unbounded set of images from across the globe. To do so, we train models on billions of random images without any data pre-processing or …
Object-Aware Background-Controlled Editing via Weighted Velocity Guidance
Carnegie Mellon University（卡内基梅隆大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Trajectory Forcing: Structure-First Generation with Controllable Semantic Trajectories
University of Tübingen（图宾根大学）；ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute（ELLIS研究所）；Tübingen AI Center（图宾根人工智能中心）
We propose an effective two-stage approach to tackle the problem of language-based Human-centric Spatio-Temporal Video Grounding (HC-STVG) task. In the first stage, we propose an Augmented 2D Temporal Adjacent Network (Augmented 2D-TAN) to temporally ground the target moment corresponding to the given description. Primarily, we improve the original 2D-TAN from two aspects: First, a temporal context-aware Bi-LSTM Aggregation Module is developed to aggregate clip-level representations, replacing the original max-pooling. Second, we propose to employ Random Concatenation Augmentation (RCA) mechan…
Object-Centric Action-Enhanced Representations for Robot Visuo-Motor Policy Learning
School of ECE, National Technical University of Athens（希腊国家技术大学电子工程学院）；Robotics Institute, Athena Research Center（阿提卡研究中心机器人研究所）；School of ECE, NTUA（希腊国家技术大学电子工程学院）；HERON - Hellenic Robotics Center of Excellence（希腊机器人卓越中心）
Columbia University（哥伦比亚大学）；University of California, Los Angeles（加州大学洛杉矶分校）
From a visual scene containing multiple people, human is able to distinguish each individual given the context descriptions about what happened before, their mental/physical states or intentions, etc. Above ability heavily relies on human-centric commonsense knowledge and reasoning. For example, if asked to identify the "person who needs healing" in an image, we need to first know that they usually have injuries or suffering expressions, then find the corresponding visual clues before finally grounding the person. We present a new commonsense task, Human-centric Commonsense Grounding, that tes…
Axial-Centric Cross-Plane Attention for 3D Medical Image Classification
National Heart Centre Singapore, Singapore（新加坡国家心脏中心）；CVS.AI, National Heart Research Institute of Singapore, Singapore（CVS.AI、新加坡国家心脏研究院）；Independent Researcher, Republic of Korea（韩国独立研究员）
Beyond Model Design: Data-Centric Training and Self-Ensemble for Gaussian Color Image Denoising
Hefei University of Technology（合肥工业大学）；Hangzhou Dianzi University（杭州电子科技大学）；Jinan University（暨南大学）；South China Agricultural University（华南农业大学）；The University of Tokyo（东京大学）
Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local …
Adept: Annotation-Denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining
College of Electrical Engineering, Zhejiang University（浙江大学电气工程学院）；School of Mechanical Engineering, Zhejiang University（浙江大学机械工程学院）；Ocean College, Zhejiang University（浙江大学海洋学院）；Hainan Institute, Zhejiang University（海南研究院）；Shanghai AI Laboratory（上海人工智能实验室）
University of Cambridge（剑桥大学）；Microsoft Research（微软研究院）
Recent years have seen a growth in user-centric applications that require effective knowledge transfer across tasks in the low-data regime. An example is personalization, where a pretrained system is adapted by learning on small amounts of labeled data belonging to a specific user. This setting requires high accuracy under low computational complexity, therefore the Pareto frontier of accuracy vs. adaptation cost plays a crucial role. In this paper we push this Pareto frontier in the few-shot image classification setting with a key contribution: a new adaptive block called Contextual Squeeze-a…
