Teaching Computers the Building Blocks of Signs
- Marshall Hurst
- 2 days ago
- 2 min read
Think about how your phone can listen to you talk and turn your words into text. Researchers want computers to do the same thing for sign language: watch a video of someone signing and recognize which signs they are making. Getting there is hard, though. A computer learns by studying huge numbers of examples, and there is far less sign language data to learn from than there is text or speech. So we have to be smart about how we teach. In a recent study, our team tried a new approach. Instead of asking a computer to memorize thousands of whole signs, we taught it the building blocks that signs are made of.
Signs are built from parts
Spoken words are built from sounds. Sign language works the same way. Every ASL sign is put together from smaller pieces: the shape of the hand, where it sits in space, how it moves, and which way the palm faces. Linguists call these pieces the phonology of the language. They are the building blocks.
We have spent years mapping these building blocks in ASL-LEX, our public database of the language. That work gave us a way to label signs by their parts, sixteen different kinds in all. In this paper we asked if a computer already had access to this map of the language, could it learn signs better?
Teaching in a smart order
We trained a model on the Sem-Lex Benchmark, a large collection of sign videos produced by deaf, fluent signers who agreed to share them. Then we tried a few ways of teaching it the sixteen building blocks.
The approach that worked best was a kind of curriculum. Rather than throwing everything at the model at once, we introduced the building blocks in an order that follows what linguists already know about how signs are structured, starting simple and building up. It is a lot like the way a good teacher sequences a lesson.
What we found
The model learned the building blocks well, getting them right about 87 percent of the time on average, and the linguistically ordered curriculum beat the other approaches. It also outperformed earlier work by a wide margin.
The bigger lesson is about how these tools should be built. When a computer treats signs as random gestures, it struggles. Giving computers real knowledge about the structure of sign language beats treating signs as gestures, and it points toward more useful tools.
Source: Kezar, L., Carlin, R., Srinivasan, T., Sehyr, Z., Caselli, N., & Thomason, J. (2023). Exploring Strategies for Modeling Sign Language Phonology. Proceedings of ESANN 2023. This post summarizes the paper for a general audience, drafted from the full paper text and abstract. Reviewed for non-deficit language.
