Application developers advertise their Apps by creating product pages with App images, and bidding on search terms. It is then crucial for App images to be highly relevant with the search terms. Solutions to this problem require an image-text matching model to predict the quality of the match between the chosen image and the search terms. In this work, we present a novel approach to matching an App image to search terms based on fine-tuning a pre-trained LXMERT model. We show that compared to the CLIP model and a baseline using a Transformer model for search terms, and a ResNet model for images, we significantly improve the matching accuracy. We evaluate our approach using two sets of labels: advertiser associated (image, search term) pairs for a given application, and human ratings for the relevance between (image, search term) pairs. Our approach achieves 0.96 AUC score for advertiser associated ground truth, outperforming the transformer+ResNet baseline and the fine-tuned CLIP model by 8% and 14%. For human labeled ground truth, our approach achieves 0.95 AUC score, outperforming the transformer+ResNet baseline and the fine-tuned CLIP model by 16% and 17%.
CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation
Institute of Trustworthy Embodied AI(可信具身人工智能研究院);Fudan University(复旦大学);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Bytedance Intelligent Creation(字节跳动智能创作)
Hollywood Town: Long-Video Generation via Cross-Modal Multi-Agent Orchestration
The Hong Kong University of Science and Technology(香港科学与技术大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of Chicago(芝加哥大学);Stanford University(斯坦福大学)
Multi-Modal Active Learning for Automatic Liver Fibrosis Diagnosis based on Ultrasound Shear Wave Elastography
Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong(香港中文大学深圳大数据研究院);Huazhong University of Science and Technology(华中科技大学);Shenzhen Third People’s Hospital(深圳市第三人民医院)
Integrating MedCLIP and Cross-Modal Fusion for Automatic Radiology Report Generation
The Bishop Strachan School(毕夏普斯特拉坎学校);Baidu Inc.(百度公司);Beihang University(北京航空航天大学);Institute of Future Education(未来教育研究所);TechArena Canada Inc.(加拿大泰克Arena公司);College of Engineering(工程学院);Northeastern University(东北大学)
AI-based Automatic Segmentation of Prostate on Multi-modality Images: A Review
Soochow University(苏州大学);Peking University First Hospital(北京大学第一医院);Duke Kunshan University(昆山杜克大学);Swinburne University of Technology(斯威本科技大学);The Hong Kong Polytechnic University(香港理工大学);UT Southwestern Medical Center(德克萨斯大学西南医学中心)
A fully 3D multi-path convolutional neural network with feature fusion and feature weighting for automatic lesion identification in brain MRI images
New Jersey Institute of Technology(新泽西理工学院);Kessler Foundation(凯斯勒基金会);Emory University(埃默里大学);Atlanta VA Medical Center(亚特兰大退伍军人事务医疗中心);Medical College of Wisconsin(威斯康星医学院);Rutgers University - Newark(罗格斯大学纽瓦克分校)
我们提出了一个全三维多路径卷积网络来预测中风病变的三维脑MRI图像。我们的多路径模型具有针对包含残余卷积块的不同模态的独立编码器、来自不同模态的加权多路径特征融合以及用于组合编码器和解码器特征的加权融合模块。与DeepMedic、3D U-Net和AnatomyNet等现有的3D CNN相比,我们的网络在220名患者的大型ATLAS基准测试中实现了最高的统计学显著交叉验证准确率(60.5%)。我们还在来自Kessler Foundation和Medical College Wisconsin的多模态图像上测试了我们的模型,并实现了65%的统计学显著交叉验证准确率,显著优于多模态3D U-Net和DeepMedic。总的来说,我们的模型提供了一个原则性的,可扩展的多路径方法,优于多通道的替代品,并在现有的基准实现高骰子精度。
HIST-AID: Leveraging Historical Patient Reports for Enhanced Multi-Modal Automatic Diagnosis
Courant Institute of Mathematical Sciences, New York University(纽约大学柯朗数学科学研究所);New York University Grossman School of Medicine(纽约大学格罗斯曼医学院);Center of Data Science, New York University(纽约大学数据科学中心);Prescient Design, Genentech(基因泰克公司Prescient Design部门);CIFAR LMB(加拿大高级研究院学习算法与机器智能实验室)