Can a Game Make Sign Language Technology More Inclusive?
- Jul 13
- 3 min read
For technology to understand sign language, it first has to learn from real examples of people signing, and it needs a lot of them. Just as important, it needs them from many different people. Sign languages vary from signer to signer, a bit like accents in spoken languages, so a tool that only learns from a handful of people ends up working well for some and poorly for everyone else. Good data has to be representative of the whole community, not just the few who happen to be nearby.
The hard part is that collecting all that video usually asks a lot of people. The typical approach is long recording sessions, signing one word after another, which is tedious work that tends to draw the same small group of volunteers each time. So we tried something different. What if contributing sign language data could actually be fun? ASL Sea Battle is a phone game, based on the classic game Battleship, and playing it quietly produces exactly the kind of labeled signing videos this technology needs. By turning the task into a game, we hoped to make it interesting to a wider range of people, so the data reflects the community more fully.
That's the idea behind ASL Sea Battle, from Danielle Bragg, Naomi Caselli, John Gallagher, Miriam Goldberg, Courtney Oka, and William Thies. It's a signing version of the classic game Battleship, built as a phone app. You play a game, and while you play, the app quietly collects real ASL videos that can help train sign language tools. You also pick up some signs along the way.
How the game works
The game follows the rules of Battleship, with a twist. On your turn, you fire at a spot on your opponent's board by recording a short video of yourself making a given sign. Your opponent then watches and names the sign to defend that spot. Two useful things happen at the same time. The recording becomes a labeled video of someone signing, and your opponent's guess becomes a second check on what that sign is. The data collection is not a separate chore. It is just part of playing.
What we found
We tested the game with twenty people, half of them deaf or hard of hearing and half hearing ASL users, and compared it against a plain recording app. The videos people made while playing were good enough to train sign language technology. Experts rated about 98 percent of them useful, and the labels players gave were accurate every single time.
The game was a little slower than the plain app, since playing takes more steps than simply hitting record. But people clearly liked it more. Almost everyone preferred the game, and many said they kept going because it was fun, not because they were asked to. That trade is worth paying attention to. A slower tool that people enjoy can end up collecting more, and more varied, data over time than a fast one they avoid.
Why it matters
The biggest challenge in sign language technology is not clever math. It is data. There is not enough of it, and what exists often comes from too few people in lab settings. A game changes who is willing to take part and how long they stay. When contributing feels like play instead of a task, more people join in, and the data starts to reflect the community more fully.
We even saw people play for different reasons. Many deaf players wanted to add variety to the data so tools would work for signers like them, while many hearing players wanted to help the technology improve. ASL Sea Battle was one of the first games built for this purpose, and the idea grew into ASL Citizen, a much larger community-driven collection of signs.
Source: Bragg, D., Caselli, N., Gallagher, J. W., Goldberg, M., Oka, C., & Thies, W. (2021). ASL Sea Battle: Gamifying Sign Language Data Collection. CHI '21. https://doi.org/10.1145/3411764.3445416. This post summarizes the paper for a general audience, drawn from the published abstract and paper. Reviewed for non-deficit language.
