Incorporating Non-Manual Features in Automatic Sign Language Tra... | Valeri Sazonov | RRS Live 2025 скачать в хорошем качестве

Incorporating Non-Manual Features in Automatic Sign Language Tra... | Valeri Sazonov | RRS Live 2025

Valeri Sazonov is a Sophomore Computer Engineering, Mathematics major in the Randall Research Scholars Program (RRSP). Their research project presentation, "Incorporating Non-Manual Features in Automatic Sign Language Translation," was completed under the advisement of Evie Malaia from the Communicative Disorders Department. Project Description: Artificial intelligence has the potential to revolutionize Augmentative and Alternative Communication (AAC) for Deaf patients, if it can provide real-time interpretation in healthcare settings. However, current sign language understanding (SLU) models remain unable to generalize to the dynamic, multi-channel, compositional nature of signed communication. This project applies interpretability methods to state-of-the-art SLU models for isolated signs and signed sentences to better understand current limitations in SLU under a linguistic framework, targeting auxilliary non-manual features. Two datasets were analyzed: a multimodal Italian Sign Language dataset combining video and radar, and a video-based Saudi Sign Language dataset.