Teaching Computers ASL With a Map of How Signs Are Built
- Marshall Hurst
- 13 hours ago
- 3 min read
Say you want to build a tool that can watch a video of someone signing and know which sign they made. The usual way to teach a computer this is to show it thousands and thousands of labeled examples. Watch enough videos of the sign CAT, and eventually the computer learns to spot CAT on its own.
But good, labeled ASL video is hard to come by. One approach is to just scrape videos off the internet, but doing that raises real questions about consent and respect for the deaf people in those videos. But that means sign language tools are often stuck learning from too little data to be as robust as tools for spoken and written languages.
In a project led by Lee Kezar, we tried a different approach. Instead of only showing the computer more examples, we gave it knowledge about how signs are put together.
The idea: sign languages have linguistic structure, so teach computers the structure
Signs aren't random shapes. Like spoken words, they're built from smaller parts. A sign has a handshape, a place on or near the body where it happens, a movement, and more. Linguists have studied these building blocks for decades. ASL-LEX, a database the Deaf Center helped build, describes thousands of signs in exactly this way.
Our idea was to hand the computer that structure directly. We built something called the American Sign Language Knowledge Graph. Think of it as a big map that connects each sign to facts about it: what handshape it uses, where it's made, what it means, how it relates to other signs. We pulled this map together from 11 different sources of knowledge about ASL, focusing on how signs are built (their phonology) and what they mean (their lexical-semantics). Then we let a computer model learn from both the videos and the map at the same time.
What they found
Giving the computer the map helped, on several different jobs.
Recognizing a single sign. When shown a video of one isolated sign, the model correctly named it 91% of the time.
Guessing the meaning of a brand-new sign. This is much harder. The model had to predict what a sign it had never been trained on might mean, just from how it's built. It got 14%. That's low on its own, but we were impressed that it was possible at all. The structure carries real clues about meaning.
Sorting real-world videos by topic. Using clips from YouTube-ASL, the model sorted videos by what they were about and got 36% right.
The cool thing is that a camputer can learn ASL better when you teach it how signs are actually built, instead of only making it memorize more examples. Knowledge can stand in for some of the data we don't have.
Why this matters
This work addresses a concrete access barrier: tools that recognize and support ASL simply don't exist at scale, because the field treats ASL as a data problem rather than as a language worthy of study from the start.
When technology is built with ASL from the ground up, drawing on decades of linguistic research done in partnership with Deaf communities, it works better and it's built with respect. This approach leans on what deaf linguists and signers already know about the language, rather than demanding an extraordinary amount of video.
Source: Kezar, L., Munikote, N., Zeng, Z., Sehyr, Z., Caselli, N., & Thomason, J. (2025). The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge. Findings of the Association for Computational Linguistics: NAACL 2025, 7032–7044. This post summarizes the paper for a general audience, drafted from the full paper text and abstract. Reviewed for non-deficit language.
