Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models
University of Trento(特伦托大学);Snap Inc.(Snap 公司);LTCI, Télécom Paris, Institut Polytechnique de Paris(巴黎高等电信学院,巴黎综合理工学院);MPI for Informatics, SIC(马克斯·普朗克信息学研究所,SIC);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会)
ACM Transactions on Graphics \c{opyright} Copyright is held by the owner/author(s) 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ACM Transactions on Graphics, http://dx.doi.org/10.1145/3635705
Neural video game simulators emerged as powerful tools to generate and edit videos. Their idea is to represent games as the evolution of an environment's state driven by the actions of its agents. While such a paradigm enables users to play a game action-by-action, its rigidity precludes more semantic forms of control. To overcome this limitation, we augment game models with prompts specified as a set of natural language actions and desired states. The result-a Promptable Game Model (PGM)-makes it possible for a user to play the game by prompting it with high- and low-level action sequences. Most captivatingly, our PGM unlocks the director's mode, where the game is played by specifying goals for the agents in the form of a prompt. This requires learning "game AI", encapsulated by our animation model, to navigate the scene using high-level constraints, play against an adversary, and devise a strategy to win a point. To render the resulting state, we use a compositional NeRF representation encapsulated in our synthesis model. To foster future research, we present newly collected, annotated and calibrated Tennis and Minecraft datasets. Our method significantly outperforms existing neural video game simulators in terms of rendering quality and unlocks applications beyond the capabilities of the current state of the art. Our framework, data, and models are available at https://snap-research.github.io/promptable-game-models/.
PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigation
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Tianjin Artificial Intelligence Innovation Center(天津人工智能创新中心);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
SpO$_2$ Predictor-Guided Stage-Wise Time-Frequency Reconstruction of Low-Quality Dual-Wavelength PPG for Oxygen Saturation Estimation
Department of Computer Science, University of California, Davis, Davis, CA, U.S.A.(加州大学戴维斯分校计算机科学系);Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, U.S.A.(加州大学戴维斯分校电气与计算机工程系)
UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(高性能计算研究所(A*STAR));Singapore Eye Research Institute, Singapore National Eye Centre(新加坡眼科研究所,新加坡国家眼科中心)
MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training
School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);Electronic Information School, Wuhan University(武汉大学电子信息学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院)
PLGSA-Transformer: Periocular Landmark-Guided Attention with Occlusion-Adaptive Cosine Thresholding for Cross-Modal Masked and Unmasked Face Recognition
Faculty of Engineering and Computer Science, Qaiwan International University(工程与计算机科学学院,卡伊万国际大学);Department of Information and Communication Technology Center (ICTC)-System Information, Ministry of Higher Education and Scientific Research(信息与通信技术中心(ICTC)-系统信息,高等教育与科学研究部)
Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
University of Lincoln, Lincoln Centre for Autonomous Systems(林肯大学,林肯自主系统中心);University of Cambridge, Institute for Manufacturing, Department of Engineering(剑桥大学,制造研究所,工程系)
Towards Interactive Lesion Segmentation in Whole-Body PET/CT with Promptable Models
German Cancer Research Center (DKFZ) Heidelberg(德国癌症研究中心(DKFZ)海德堡);Division of Medical Image Computing(医学影像计算 division);HIDSS4Health - Helmholtz Information and Data Science School for Health(HIDSS4Health - 欧文信息与数据科学健康学校);Faculty of Mathematics and Computer Science(数学与计算机科学学院);Heidelberg University(海德堡大学);Helmholtz Imaging(海德堡成像);Pattern Analysis and Learning Group(模式分析与学习组);Department of Radiation Oncology(放射肿瘤科)
SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders
Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);JD Explore Academy(京东探索研究院);Renmin University of China(中国人民大学)
What to Hide from Your Students: Attention-Guided Masked Image Modeling
National Technical University of Athens(雅典国立技术大学);Institute of Advanced Research in Artificial Intelligence (IARAI)(人工智能高级研究院(IARAI));Athena RC(雅典娜研究中心);University of Crete(克里特大学);IACM-Forth(希腊计算机科学研究所-基础研究与技术基金会)
Promptable segmentation with region exploration enables minimal-effort expert-level prostate cancer delineation
UCL Hawkes Institute(UCL Hawkes Institute;医学物理与生物医学工程系);Department of Medical Physics and Biomedical Engineering(大学学院伦敦);University College London(医学成像中心);Centre of Medical Imaging(肿瘤学系);Department of Oncology(阿迦·柯尔大学医院);Aga Khan University Hospital(生物工程中心;工程与材料科学学院);Centre for Bioengineering;School of Engineering and Materials Science