JurisTCU：一个带有查询相关性判断的巴西葡萄牙语信息检索数据集
JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgments
AI总结：
本文介绍了一个包含16045份巴西联邦会计法院判例文书和150个带有相关性标注查询的巴西葡萄牙语信息检索数据集，通过混合方法产生相关性判断，展示了文档扩展方法在提升BM25性能上的显著效果，以及OpenAI模型在短关键词查询中的优越表现。
摘要：
本文介绍了一个包含16045份巴西联邦会计法院判例文书和150个带有相关性标注查询的巴西葡萄牙语信息检索数据集，通过混合方法产生相关性判断，展示了文档扩展方法在提升BM25性能上的显著效果，以及OpenAI模型在短关键词查询中的优越表现。
英文摘要：
This paper introduces JurisTCU, a Brazilian Portuguese dataset for legal information retrieval (LIR). The dataset is freely available and consists of 16,045 jurisprudential documents from the Brazilian Federal Court of Accounts, along with 150 queries annotated with relevance judgments. It addresses the scarcity of Portuguese-language LIR datasets with query relevance annotations. The queries are organized into three groups: real user keyword-based queries, synthetic keyword-based queries, and synthetic question-based queries. Relevance judgments were produced through a hybrid approach combining LLM-based scoring with expert domain validation. We used JurisTCU in 14 experiments using lexical search (document expansion methods) and semantic search (BERT-based and OpenAI embeddings). We show that the document expansion methods significantly improve the performance of standard BM25 search on this dataset, with improvements exceeding 45% in P@10, R@10, and nDCG@10 metrics when evaluating short keyword-based queries. Among the embedding models, the OpenAI models produced the best results, with improvements of approximately 70% in P@10, R@10, and nDCG@10 metrics for short keyword-based queries, suggesting that these dense embeddings capture semantic relationships in this domain, surpassing the reliance on lexical terms. Besides offering a dataset for the Portuguese-language IR research community, suitable for evaluating search systems, the results also contribute to enhancing a search system highly relevant to Brazilian citizens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
