We introduce Unimotion, the first unified multi-task human motion model capable of both flexible motion control and frame-level motion understanding. While existing works control avatar motion with global text conditioning, or with fine-grained per frame scripts, none can do both at once. In addition, none of the existing works can output frame-level text paired with the generated poses. In contrast, Unimotion allows to control motion with global text, or local frame-level text, or both at once, providing more flexible control for users. Importantly, Unimotion is the first model which by design outputs local text paired with the generated poses, allowing users to know what motion happens and when, which is necessary for a wide range of applications. We show Unimotion opens up new applications: 1.) Hierarchical control, allowing users to specify motion at different levels of detail, 2.) Obtaining motion text descriptions for existing MoCap data or YouTube videos 3.) Allowing for editability, generating motion from text, and editing the motion via text edits. Moreover, Unimotion attains state-of-the-art results for the frame-level text-to-motion task on the established HumanML3D dataset. The pre-trained model and code are available available on our project page at https://coral79.github.io/uni-motion/.
UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis
Shanghai Artificial Intelligence Laboratory;Shanghai Innovation Institute;Shanghai Jiao Tong University;Shanghai Institute of Optics;Fudan University;University of Cambridge;Monash University;DAMO Academy, Alibaba Group;Imperial College London;The University of Hong Kong;The Hong Kong University of Science;Hupan Lab;The Chinese University of Hong Kong
Unifying Scale-Aware Depth Prediction and Perceptual Priors for Monocular Endoscope Pose Estimation and Tissue Reconstruction
The Robotics and Mechatronics Group, University of Twente(代尔夫嫩兰道5号,埃因霍温大学机器人与机电组);The Department of Surgical Oncology, The Netherlands Cancer Institute-Antoni van Leeuwenhoek(荷兰癌症研究所-安托万·李文霍克医院外科肿瘤科)
MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
MedVisAI Lab(MedVisAI实验室);ECE, Northwestern University(电子工程系,西北大学);Institute for Infocomm Research (I 2 R), A*STAR, Singapore(信息与通信研究所(I2R),A*STAR,新加坡);Lee Kong Chian School of Medicine, Nanyang Technological University(Lee Kong Chian医学院,南洋理工大学)