信息抽取是否存在“一模型适配所有”的方法？重新审视任务定义偏差

重新审视语言模型中的不确定性量化评估：与响应长度的虚假交互对结果产生偏差

Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results

本文揭示了UQ方法和正确性函数间的相互长度偏差会扭曲语言模型不确定性量化的AUROC评估，并发现LM-as-a-judge方法受长度偏差影响最小，是实现公平UQ评估的有前景路径。

Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks

Syntactic rules in natural language typically need to make reference to hierarchical sentence structure. However, the simple examples that language learners receive are often equally compatible with linear rules. Children consistently ignore these linear explanations and settle instead on the correct hierarchical one. This fact has motivated the proposal that the learner's hypothesis space is constrained to include only hierarchical rules. We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way. We simulate the acquisition of question formation,…

Revisiting Few-sample BERT Fine-tuning

This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instabili…

Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases

Previous literatures show that pre-trained masked language models (MLMs) such as BERT can achieve competitive factual knowledge extraction performance on some datasets, indicating that MLMs can potentially be a reliable knowledge source. In this paper, we conduct a rigorous study to explore the underlying predicting mechanisms of MLMs over different extraction paradigms. By investigating the behaviors of MLMs, we find that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts. Furthermore, incorporating illustrative cases and external contexts improve kn…

Chinchilla方法2的问题2：等效FLOP拟合中的系统性偏差

Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

本文研究了Chinchilla方法2在拟合神经网络扩展定律时的系统性偏差，指出其在无噪声合成数据中仍存在偏差，影响参数分配，并提出通过变量投影方法改进。

超越一刀切：用于高效NLG评估提示的反向学习

Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts

针对NLG评估提示敏感性问题，提出反向学习方法，自动生成模型特定评估提示，仅需单样本，提升效率与鲁棒性。