The sound effects that designers add to videos are designed to convey a particular artistic effect and, thus, may be quite different from a scene's true sound. Inspired by the challenges of creating a soundtrack for a video that differs from its true sound, but that nonetheless matches the actions occurring on screen, we propose the problem of conditional Foley. We present the following contributions to address this problem. First, we propose a pretext task for training our model to predict sound for an input video clip using a conditional audio-visual clip sampled from another time within the same source video. Second, we propose a model for generating a soundtrack for a silent input video, given a user-supplied example that specifies what the video should "sound like". We show through human studies and automated evaluation metrics that our model successfully generates sound from video, while varying its output according to the content of a supplied example. Project site: https://xypb.github.io/CondFoleyGen/
MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation
Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tianjin University(天津大学);Kuaishou Technology Co., Ltd(快手科技有限公司);Department of Automation, Tsinghua University(清华大学自动化系);Beijing National Research Center for Information Science and Technology, Tsinghua University(北京信息科学与技术国家研究中心(清华大学));University of Chinese Academy of Sciences(中国科学院大学);Shanghai Polytechnic University(上海第二工业大学)
FoleyDesigner: Immersive Stereo Foley Generation with Precise Spatio-Temporal Alignment for Film Clips
Shanghai Film Academy, Shanghai University(上海大学上海电影学院);Shanghai Engineering Research Center of Motion Picture Special Effects(上海电影特效工程技术研究中心);University of Surrey, UK(英国萨里大学)
Tri-Ergon: Fine-grained Video-to-Audio Generation with Multi-modal Conditions and LUFS Control
vivo Mobile Communication Co., Ltd(维沃移动通信有限公司);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Northwestern Polytechnical University(西北工业大学);OpenNLPLab
Language-Guided Music Recommendation for Video via Prompt Analogies
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Adobe Research(奥多比研究中心);Czech Institute of Informatics, Robotics and Cybernetics(捷克信息学、机器人学与控制论研究所);Czech Technical University(捷克技术大学)