当良好且可复现的结果是泥足巨人:软件质量在NLP中的重要性
When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP
AI总结:
本文通过修复Conformer实现中的三个错误,揭示良好可复现结果仍可能掩盖代码缺陷并误导研究,进而提出代码质量检查清单并发布测试库pangoliNN以提升NLP软件质量。
摘要:
尽管代码正确性在研究实验中起着至关重要的作用,但人们往往仅根据结果的感知质量来推定代码正确性。这种假设存在产生错误结果和可能具有误导性发现的风险。为了解决这一问题,我们认为当前对可复现性的关注应与对软件质量的重视齐头并进。我们展示了一项案例研究,在其中我们识别并修复了最先进的Conformer架构的广泛使用实现中的三个错误。通过在多种语言上的语音识别和翻译实验,我们证明错误的存在并不妨碍获得良好且可复现的结果,然而这些结果可能导致不正确的结论,从而可能误导未来的研究。作为对策,我们提出了一份代码质量检查清单,并发布了pangoliNN,一个专门用于测试神经模型的库,旨在促进编码最佳实践并提高NLP社区内的研究软件质量。
英文摘要:
Despite its crucial role in research experiments, code correctness is often presumed only on the basis of the perceived quality of results. This assumption comes with the risk of erroneous outcomes and potentially misleading findings. To address this issue, we posit that the current focus on reproducibility should go hand in hand with the emphasis on software quality. We present a case study in which we identify and fix three bugs in widely used implementations of the state-of-the-art Conformer architecture. Through experiments on speech recognition and translation in various languages, we demonstrate that the presence of bugs does not prevent the achievement of good and reproducible results, which however can lead to incorrect conclusions that potentially misguide future research. As a countermeasure, we propose a Code-quality Checklist and release pangoliNN, a library dedicated to testing neural models, with the goal of promoting coding best practices and improving research software quality within the NLP community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。