In the rapidly advancing field of multi-modal machine learning (MMML), the convergence of multiple data modalities has the potential to reshape various applications. This paper presents a comprehensive overview of the current state, advancements, and challenges of MMML within the sphere of engineering design. The review begins with a deep dive into five fundamental concepts of MMML:multi-modal information representation, fusion, alignment, translation, and co-learning. Following this, we explore the cutting-edge applications of MMML, placing a particular emphasis on tasks pertinent to engineering design, such as cross-modal synthesis, multi-modal prediction, and cross-modal information retrieval. Through this comprehensive overview, we highlight the inherent challenges in adopting MMML in engineering design, and proffer potential directions for future research. To spur on the continued evolution of MMML in engineering design, we advocate for concentrated efforts to construct extensive multi-modal design datasets, develop effective data-driven MMML techniques tailored to design applications, and enhance the scalability and interpretability of MMML models. MMML models, as the next generation of intelligent design tools, hold a promising future to impact how products are designed.
Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews
University of British Columbia（不列颠哥伦比亚大学）
内容：用户在应用商店或社交媒体上撰写的移动应用评论是应用开发人员的重要资源。分析应用评论已被证明对软件工程的许多领域都很有用（例如，需求工程、测试）。应用评论的自动分类需要大量的工作来手动管理标记的数据集。当分类目的发生变化时（例如，识别错误与可用性问题或情感），新的数据集应该被标记，这会阻止开发的模型在实践中用于新的期望类/任务的可扩展性。最近的预训练神经语言模型（PTM）以无监督的方式在大型语料库上进行训练，并成功解决了类似的自然语言处理问题。然而，PTM的适用性没有探索应用程序审查分类目标：我们调查的好处PTM应用程序审查分类相比，现有的模型，以及PTM在多个设置的可转移性。方法：我们实证研究的准确性和时间效率的PTM相比，以前的方法，使用六个数据集从文献中。此外，我们还研究了在应用程序评论上训练的PTM（即特定于域的PTM）的性能。我们建立了不同的研究来评估多个设置中的PTM：二元与多类分类，zero-shot分类（当新标签引入模型时），多任务设置，以及来自不同资源的评论分类。这些数据集是来自Google Play Store、Apple App Store和Twitter数据的手动标记的应用程序评论数据集。在所有情况下，将使用微观和宏观精度，召回率和F1分数，我们将报告模型训练和预测所需的时间。
PhytoSynth: Leveraging Multi-modal Generative Models for Crop Disease Data Generation with Novel Benchmarking and Prompt Engineering Approach
Gulf Coast Research and Education Center (GCREC), University of Florida（佛罗里达大学 Gulf Coast Research and Education Center）；Citrus Research and Education Center (CREC), University of Florida（佛罗里达大学 Citrus Research and Education Center）
A Survey of Automatic Prompt Engineering: An Optimization Perspective
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；East China Normal University（华东师范大学）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）
TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineering
Toursun Synbio（拓荪合成生物）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Cambridge（剑桥大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；City University of Hong Kong（香港城市大学）
Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices
Geneva University Hospitals（日内瓦大学医院）；University of Geneva（日内瓦大学）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；Sorbonne Université（索邦大学）；INSERM（法国国家健康与医学研究院）；Université Sorbonne Paris-Nord（索邦巴黎北大学）
