LipLearner：移动设备上可定制的无声语音交互
LipLearner: Customizable Silent Speech Interactions on Mobile Devices
摘要：
无声语音接口是一种很有前途的技术，它可以实现自然语言的私人通信。然而，以前的方法只支持一个小的和不灵活的词汇表，这导致有限的表达能力。我们利用对比学习来学习有效的唇读表示，从而以最小的用户努力实现Few-Shot命令自定义。我们的模型在野外数据集上对不同的光照、姿势和手势条件表现出很高的鲁棒性。对于25个命令分类，仅使用一次射击就可以实现0.8947的F1分数，并且可以通过从更多数据中自适应学习来进一步提高其性能。这种普遍性使我们能够开发一个移动无声语音界面，该界面具有设备上的微调和视觉关键字识别功能。一项用户研究表明，使用LipLearner，用户可以定义自己的命令，并通过在线增量学习方案保证高可靠性。主观反馈表明，我们的系统提供了必要的功能，可定制的无声语音交互具有高可用性和可学习性。
英文摘要：
Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
