This study is dedicated to assessing the capabilities of large language models (LLMs) such as GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo in extracting structured information from scientific documents in materials science. To this end, we primarily focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. Due to the evident lack of datasets within Materials Informatics (MI), we evaluated using SuperMat, based on superconductor research, and MeasEval, a generic measurement evaluation corpus. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline). We introduce a novel methodology for the comparative analysis of intricate material expressions, emphasising the standardisation of chemical formulas to tackle the complexities inherent in materials science information assessment. For NER, LLMs fail to outperform the baseline with zero-shot prompting and exhibit only limited improvement with few-shot prompting. However, a GPT-3.5-Turbo fine-tuned with the appropriate strategy for RE outperforms all models, including the baseline. Without any fine-tuning, GPT-4 and GPT-4-Turbo display remarkable reasoning and relationship extraction capabilities after being provided with merely a couple of examples, surpassing the baseline. Overall, the results suggest that although LLMs demonstrate relevant reasoning skills in connecting concepts, specialised models are currently a better choice for tasks requiring extracting complex domain-specific entities like materials. These insights provide initial guidance applicable to other materials science sub-domains in future work.
Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining
UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales(新南威尔士大学);School of Electrical Engineering and Computer Science, The University of Queensland(昆士兰大学);Microsoft Copilot Studio AI(微软Copilot Studio AI);Departments of Mathematics & Department of Computer and Information Science, University of Pennsylvania(宾夕法尼亚大学);Department of Civil Engineering, The University of Hong Kong(香港大学)
A Systematic Literature Review about Idea Mining: The Use of Machine-driven Analytics to Generate Ideas
Stockholm University(斯德哥尔摩大学)
创意产生是创新的核心活动。数字数据源是创新的来源,如专利、出版物、社交媒体、网站等,正以前所未有的速度增长手动生成想法是耗时的,并且受到所涉及的个人的主观性的影响。因此,使用机器驱动的数据分析技术来分析数据以产生想法,并通过服务用户来支持想法的产生是有用的。本研究的目的是研究最先进的机器驱动分析的想法生成和数据源,因此本研究的结果通常将作为选择技术和数据源的指导方针。进行了系统的文献综述,以识别来自IEEE、Scopus、Web of Science和Google Scholar的相关学术文献。我们选择了71篇文章,并对它们进行了主题分析。本研究的结果表明,通过机器驱动分析的想法生成应用文本挖掘,信息检索(IR),人工智能(AI),深度学习,机器学习,统计技术,自然语言处理(NLP),基于NLP的形态分析,网络分析和文献计量学来支持想法生成。结果包括通过机器驱动的想法分析产生想法的技术和程序列表。此外,在想法生成中使用的表征和算法进行了总结。今后,可以探索旨在产生想法的工具。
A foundation model for human-AI collaboration in medical literature mining
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health(美国国立卫生研究院国家糖尿病、消化和肾脏疾病研究所);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);National Eye Institute, National Institutes of Health(美国国立卫生研究院国家眼科研究所);Northwestern University Feinberg School of Medicine(西北大学费恩伯格医学院);Chungbuk National University Hospital(忠北国立大学医院);Chungbuk National University College of Medicine(忠北国立大学医学院);University of Pittsburgh Medical Center(匹兹堡大学医学中心);Weill Cornell Medicine(威尔康奈尔医学院);UT Southwestern Medical Center(德克萨斯大学西南医学中心);University of Washington(华盛顿大学);NYU Langone Health(纽约大学朗格尼健康中心);OSF HealthCare Illinois Neurological Institute(OSF医疗保健伊利诺伊神经研究所)
Harnessing X-ray Absorption Spectroscopy Data through Multimodal Mining of Battery Literature
Argonne National Laboratory(阿贡国家实验室);NVIDIA(英伟达);University of California, Berkeley(加州大学伯克利分校);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);The University of Chicago(芝加哥大学)
COVID-19 therapy target discovery with context-aware literature mining
Jožef Stefan International Postgraduate School(约瑟夫·斯特凡国际研究生院);Jožef Stefan Institute(约瑟夫·斯特凡研究所);University of Ljubljana(卢布尔雅那大学);University of Nova Gorica(新戈里察大学);Temida d.o.o(Temida有限责任公司)
AHAM: Adapt, Help, Ask, Model -- Harvesting LLMs for literature mining
Jožef Stefan Institute(约瑟夫·斯特凡研究所);International Postgraduate School Jožef Stefan(约瑟夫·斯特凡国际研究生院);University of Nova Gorica(新戈里察大学);Temida d.o.o(Temida有限责任公司);University of Ljubljana(卢布尔雅那大学)
在本文中,我们解决了从专利中提取科学参考文献的挑战。我们将这个问题作为一个序列标记任务来处理,并研究了BERT模型在提取这些长序列方面的优点。专利中对科学文献的引用与研究科学与工业之间的联系有关。大多数先前的工作只使用这种分析的头版引文,这些引文在专利档案的元数据中提供。在本文中,我们建立在以前的工作,使用条件随机场(CRF)和天赋的参考提取。我们提高了训练数据的质量,并在标记数据上训练了三个基于BERT的模型(BERT,bioBERT,sciBERT)。我们发现,改进的训练数据导致训练模型的质量有很大的提高。此外,BERT模型击败了CRF和Flair,通过交叉验证获得了约97%的召回分数。使用最佳模型,我们标记了大量的33000项专利,提取引用,并将它们与Web of Science数据库中的出版物进行匹配。我们提取了比旧的训练数据和方法多50%的参考文献:总共73.5万个参考文献。有了这些专利-出版链接,后续研究将进一步分析哪些类型的科学工作导致发明。