ReHEarSSE | Xuefu Dong

(a) ReHEarSSE uses a novel earbud-based ultrasonic sensing method to infer silently spelled words, even if they are not in the training lexicon. (b) ReHEarSSE can be used while interacting with an extended-reality device for hands-free text input. (c) ReHEarSSE can also be used on-the-go while users’ hands are unavailable or inconvenient for text entry on a smartwatch or smart eyewear

Silent speech interaction (SSI) allows users to discreetly input text without using their hands. Existing wearable SSI systems typically require custom devices and are limited to a small lexicon, limiting their utility to a small set of command words. This work proposes ReHEarSSE, an earbud-based ultrasonic SSI system capable of generalizing to words that do not appear in its training dataset, providing support for nearly an entire dictionary’s worth of words (Dong et al., 2024). As a user silently spells words, ReHEarSSE uses autoregressive features to identify subtle changes in ear canal shape. ReHEarSSE infers words using a deep learning model trained to optimize connectionist temporal classification (CTC) loss with an intermediate embedding that accounts for different letters and transitions between them. We find that ReHEarSSE recognizes unseen words with an accuracy of 89.3 ± 10.9%.

References

2024

ReHEarSSE: Recognizing Hidden-in-the-Ear Silently Spelled Expressions

Xuefu Dong, Yifei Chen , Yuuki Nishiyama , and 4 more authors

In Proceedings of the CHI Conference on Human Factors in Computing Systems , 2024

Acceptance Rate 26.3%

Abs PDF Video

Silent speech interaction (SSI) allows users to discreetly input text without using their hands. Existing wearable SSI systems typically require custom devices and are limited to a small lexicon, limiting their utility to a small set of command words. This work proposes ReHEarSSE, an earbud-based ultrasonic SSI system capable of generalizing to words that do not appear in its training dataset, providing support for nearly an entire dictionary’s worth of words. As a user silently spells words, ReHEarSSE uses autoregressive features to identify subtle changes in ear canal shape. ReHEarSSE infers words using a deep learning model trained to optimize connectionist temporal classification (CTC) loss with an intermediate embedding that accounts for different letters and transitions between them. We find that ReHEarSSE recognizes 100 unseen words with an accuracy of 89.3%.