Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis
School of Computing, University of Georgia(佐治亚大学计算学院);School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院);School of Data Science, University of Virginia(弗吉尼亚大学数据科学学院);University of Georgia College of Pharmacy(佐治亚大学药学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
Shaochen Xu, Zihao Wu, Huaqin Zhao, Peng Shu, Zhengliang Liu, Wenxiong Liao, Sheng Li, Andrea Sikora, Tianming Liu, Xiang Li
In this study, we leverage LLM to enhance the semantic analysis and develop similarity metrics for texts, addressing the limitations of traditional unsupervised NLP metrics like ROUGE and BLEU. We develop a framework where LLMs such as GPT-4 are employed for zero-shot text identification and label generation for radiology reports, where the labels are then used as measurements for text similarity. By testing the proposed framework on the MIMIC data, we find that GPT-4 generated labels can significantly improve the semantic similarity assessment, with scores more closely aligned with clinical ground truth than traditional NLP metrics. Our work demonstrates the possibility of conducting semantic analysis of the text data using semi-quantitative reasoning results by the LLMs for highly specialized domains. While the framework is implemented for radiology report similarity analysis, its concept can be extended to other specialized domains as well.
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
Dept. of Computer Science and Engg., IIT Kharagpur(印度Kharagpur理工学院计算机科学与工程系);Dept. of Medicine (Biomedical Informatics), Stanford University(斯坦福大学医学院(生物医学信息学))
Specialized Document Embeddings for Aspect-based Similarity of Research Papers
DFKI GmbH(德国人工智能研究中心);Ernst & Young GmbH WPG(安永会计师事务所(德国));University of Wuppertal(伍珀塔尔大学);University of Göttingen(哥廷根大学)
文档嵌入和相似性度量支持基于内容的推荐系统,其中文档通常表示为单个通用嵌入。然而,在单个向量表示上计算的相似性仅提供了关于文档相似性的一个视角,忽略了哪些方面使两个文档相似。为了解决这一限制,已经开发了基于方面的相似性措施,使用文档分割或成对多类文档分类。虽然分割损害了文档的连贯性,但成对分类方法对大规模语料库的扩展性很差。在本文中,我们把基于方面的相似性作为一个经典的向量相似性问题,在特定方面的嵌入空间。我们不是将文档表示为单个通用嵌入,而是表示为多个专用嵌入。我们的方法避免了文件分割和规模线性w.r.t.语料库的大小。在实证研究中,我们使用包含157,606篇研究论文的Papers with Code语料库,并将各自研究论文的任务,方法和数据集作为其方面。我们比较和分析三个通用的文档嵌入,六个专门的文档嵌入和成对的分类基线的背景下,研究论文的建议。作为通用文档嵌入,我们考虑FastText,SciBERT和SPECTER。为了计算专门的文档嵌入,我们比较了三种替代方法,灵感来自改造,微调和暹罗网络。在我们的实验中,Siamese SciBERT获得了最高的分数。额外的分析表明,对数据集方面和对每个研究论文的方法方面的通用文档嵌入的隐含偏见。我们的方法基于方面的文档嵌入减轻潜在的风险所产生的隐性偏见,使他们明确。
Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selection
Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology(哈尔滨工业大学社会计算与信息检索研究中心);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Du Xiaoman Technology (Beijing) Co., Ltd.(度小满科技(北京)有限公司)