评估大语言模型在真实世界消费设备维修问题上的有效性
Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questions
AI总结：
本文通过引入包含991个真实维修问题的基准测试，评估六种大语言模型在英语和孟加拉语上的正确性、完整性、实用性和安全性，发现模型虽能提供有用帮助，但在高风险维修任务中仍不可靠。
摘要：
消费设备维修是大语言模型（LLMs）一个重要但尚未充分探索的测试平台。维修任务需要对不完整的问题描述、特定硬件的诊断、可操作的故障排除和安全关键决策进行推理，其中错误的建议可能导致设备损坏、电池危险或永久性数据丢失。我们引入了一个包含991个来自Reddit的真实世界维修问题的基准测试，涵盖手机维修、电脑维修和数据恢复，每个问题都配有技术人员编写的参考解决方案，并提供孟加拉语翻译以评估跨语言性能。我们使用四个维修特定标准（正确性、完整性、实用性和安全性）评估了六种最先进的LLMs在英语和孟加拉语上的表现。我们的结果表明，虽然LLMs可以提供有用的维修帮助，但在没有严格评估和明确安全保护措施的情况下，它们在高风险的真实世界维修任务中仍然不可靠。手机维修是最困难且对安全最敏感的领域，所有模型在板级诊断、维修优先级排序和安全恢复程序方面都犯了重大错误。跨领域和模型，孟加拉语回答的表现始终低于英语回答。在评估的模型中，GPT-5.4整体表现最佳。
英文摘要：
Consumer device repair is an important but underexplored testbed for large language models (LLMs). Repair tasks require reasoning over incomplete problem descriptions, hardware-specific diagnostics, actionable troubleshooting, and safety-critical decisions, where incorrect advice can cause device damage, battery hazards, or permanent data loss. We introduce a benchmark of 991 real-world repair questions from Reddit spanning phone repair, computer repair, and data recovery, each paired with technician-written reference solutions, and provide Bangla translations to evaluate cross-lingual performance. We evaluate six state-of-the-art LLMs in English and Bangla using four repair-specific criteria: correctness, completeness, practicality, and safety. Our results show that while LLMs can provide useful repair assistance, they remain unreliable for high-risk real-world repair tasks without rigorous evaluation and explicit safety safeguards. Phone repair is the most difficult and safety-sensitive domain, and all models make substantial errors in board-level diagnosis, repair prioritization, and safe recovery procedures. Across domains and models, Bangla responses consistently perform worse than English responses. Among the evaluated models, GPT-5.4 performs best overall.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
