This paper introduces the first text-guided work for generating the sequence of hand-object interaction in 3D. The main challenge arises from the lack of labeled data where existing ground-truth datasets are nowhere near generalizable in interaction type and object category, which inhibits the modeling of diverse 3D hand-object interaction with the correct physical implication (e.g., contacts and semantics) from text prompts. To address this challenge, we propose to decompose the interaction generation task into two subtasks: hand-object contact generation; and hand-object motion generation. For contact generation, a VAE-based network takes as input a text and an object mesh, and generates the probability of contacts between the surfaces of hands and the object during the interaction. The network learns a variety of local geometry structure of diverse objects that is independent of the objects' category, and thus, it is applicable to general objects. For motion generation, a Transformer-based diffusion model utilizes this 3D contact map as a strong prior for generating physically plausible hand-object motion as a function of text prompts by learning from the augmented labeled dataset; where we annotate text labels from many existing 3D hand and object motion data. Finally, we further introduce a hand refiner module that minimizes the distance between the object surface and hand joints to improve the temporal stability of the object-hand contacts and to suppress the penetration artifacts. In the experiments, we demonstrate that our method can generate more realistic and diverse interactions compared to other baseline methods. We also show that our method is applicable to unseen objects. We will release our model and newly labeled data as a strong foundation for future research. Codes and data are available in: https://github.com/JunukCha/Text2HOI.
3D Hand Mesh-Guided AI-Generated Malformed Hand Refinement with Hand Pose Transformation via Diffusion Model
Department of Computer and Information Science, University of Macau, Macau, China(计算机与信息科学系,澳门大学,澳门,中国);ShanghaiTech University, Shanghai, China(上海科技大学,上海,中国)
Institute of Automation Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
场景文本检测任务由于其广泛的应用而在计算机视觉中引起了相当大的关注。近年来,许多研究者将语义分割的方法引入到场景文本检测中,取得了很好的效果。为了提高场景文本检测的分割效果,提出了一种基于条件生成对抗网络的场景文本检测器框架--DGST(Discriminator Guided Scene Text detector)。该方法取代了现有的基于语义分割的二值文本评分图,生成了包含更多信息的多尺度软文本评分图,更合理地表示文本位置,解决了文本提取过程中文本像素粘连的问题。标准数据集上的实验表明,所提出的DGST带来了显着的增益,并优于国家的最先进的方法。具体而言,它在ICDAR 2015数据集上实现了87%的F-测量。
CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility
The Chinese University of Hong Kong(香港中文大学);International Digital Economy Academy (IDEA)(国际数字经济学院);The University of Hong Kong(香港大学);Tsinghua University(清华大学)
FAGStyle: Feature Augmentation on Geodesic Surface for Zero-shot Text-guided Diffusion Image Style Transfer
School of Computer Engineering and Science, Shanghai University(上海大学计算机工程与科学学院);Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education(教育部硅酸盐文物保护重点实验室(上海大学))
RSEdit: Text-Guided Image Editing for Remote Sensing
School of Earth Sciences, Zhejiang University(浙江大学地球科学学院);Zhejiang Provincial Key Laboratory of Geographic Information Science(浙江省地理信息科学重点实验室);Key Laboratory of Spatio-temporal Information and Intelligent Services (LSIIS), Ministry of Natural Resources of the People’s Republic of China(中华人民共和国自然资源部时空信息与智能服务重点实验室);ReLER, CCAI, Zhejiang University(ReLER,中国人工智能学会,浙江大学)
Text Guided Image Editing with Automatic Concept Locating and Forgetting
King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Chinese Academy of Sciences(中国科学院);Sun Yat-sen University(中山大学);University of Auckland(奥克兰大学);University of Missouri(密苏里大学)
提出Locate and Forget (LaF)方法,通过比较句法树自动定位图像中待修改目标概念并遗忘其存在线索,解决了文本引导图像编辑中物体属性残留问题。
Towards Open-World Text-Guided Face Image Generation and Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);University College London(伦敦大学学院);Chinese University of Hongkong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳市大数据研究院)