让过去动起来:通过视频生成重建三叶虫
Animating the Past: Reconstruct Trilobite via Video Generation
AI总结:
本文提出一种自动文本到视频提示学习方法,利用大型语言模型生成提示并基于视觉真实感与流畅性奖励进行训练,结合9088张三叶虫化石图像微调视频生成模型,实现了高真实感的三叶虫行为动态重建。
摘要:
古生物学研究过去生命,从根本上依赖化石来重建古代生态系统并理解进化动态。三叶虫作为一类重要的已灭绝海洋节肢动物,通过其保存完好的化石记录为古生代环境提供了宝贵见解。从静态化石重建三叶虫行为将为科学研究和教育中的动态重建设立新标准。尽管具有潜力,当前用于此目的的计算方法(如文本到视频 T2V)面临重大挑战,例如保持视觉真实感和一致性,这阻碍了它们在科学语境中的应用。为克服这些障碍,我们引入了一种自动 T2V 提示学习方法。在该框架中,用于微调视频生成模型的提示由大型语言模型生成,该模型使用量化生成视频视觉真实感和流畅性的奖励进行训练。视频生成模型的微调以及奖励计算利用了收集的 9,088 张 Eoredlichia intermedia 化石图像数据集,该数据集提供了所有三叶虫类别视觉细节的共同代表。定性和定量实验表明,与强大的基线相比,我们的方法能够生成视觉真实感显著更高的三叶虫视频,有望促进科学理解和公众参与。
英文摘要:
Paleontology, the study of past life, fundamentally relies on fossils to reconstruct ancient ecosystems and understand evolutionary dynamics. Trilobites, as an important group of extinct marine arthropods, offer valuable insights into Paleozoic environments through their well-preserved fossil records. Reconstructing trilobite behaviour from static fossils will set new standards for dynamic reconstructions in scientific research and education. Despite the potential, current computational methods for this purpose like text-to-video (T2V) face significant challenges, such as maintaining visual realism and consistency, which hinder their application in science contexts. To overcome these obstacles, we introduce an automatic T2V prompt learning method. Within this framework, prompts for a fine-tuned video generation model are generated by a large language model, which is trained using rewards that quantify the visual realism and smoothness of the generated video. The fine-tuning of the video generation model, along with the reward calculations make use of a collected dataset of 9,088 Eoredlichia intermedia fossil images, which provides a common representative of visual details of all class of trilobites. Qualitative and quantitative experiments show that our method can generate trilobite videos with significantly higher visual realism compared to powerful baselines, promising to boost both scientific understanding and public engagement.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。