Putting sign language AI into users’ hands

Google DeepMind’s SL2T model introduces a new capability for sign-language-to-text translation, targeting Deaf and hard-of-hearing users. The model is deployed in Gboard and Live Transcribe on Pixel 11 devices, allowing users to sign instead of type for tasks like web searches, messaging, or conversational responses. This marks the first time sign-language AI has moved from research to consumer products, addressing a long-standing gap in accessibility tools for non-spoken languages.

The technical approach differs from earlier attempts, such as sign-language gloves, which failed to account for the full linguistic complexity of sign languages. SL2T treats sign languages as independent languages with their own grammars and lexicons, requiring true machine translation rather than sequential word mapping. The model processes sign language as a sequence of pose landmarks, tracked by an on-device model (MediaPipe Holistic), which extracts geometric coordinates from video feeds. This method avoids privacy concerns by discarding raw video immediately after processing.

SL2T’s training data includes over 100,000 hours of sign language footage across more than 50 languages, with a focus on American Sign Language (ASL). Training on diverse languages and dialects improves the model’s ability to generalize, outperforming single-language models. However, the system’s reliance on pose tracking means it may struggle with occlusions or low-light conditions, where landmark detection becomes unreliable. The initial rollout is limited to Pixel 11 devices and ASL-to-English translation, with broader device and language support planned for the future.

For engineers, SL2T provides a reference implementation for integrating sign-language translation into applications. The model’s direct translation approach avoids the limitations of gloss-based systems, which often miss non-manual markers and spatial constructions. However, the computational demands of pose tracking and translation may limit deployment to high-end devices initially. The privacy-preserving design, which discards raw video, could serve as a template for other vision-based assistive technologies, though it also introduces dependencies on MediaPipe Holistic’s accuracy and performance.