Sem-Lex: A Dataset of Isolated Signs
- Oct 3, 2024
- 2 min read
Updated: Jul 17
Sign language technology could expand access for deaf people, but only if it is built well. One of the main barriers to high quality sign language technology is high quality, large-scale datasets.
That is in part because datasets are hard to create. Curated datasets are often really small, which makes them insufficient for many applications. Another approach is to scrape data off the internet, but this generally means the signers' do not consent to be included in the dataset and there isn't a record of who those signers are. Scraped data are also often quite messy.
We decided to build a curated dataset that is large enough to train computers to recognize isolated signs. We chose isolated sign recognition as the task because it was an easier task to teach computers than recognizing full sentences, and because there are interesting applications (e.g., a dictionary search).
We built Sem-Lex by showing participants a cue sign, and asking them to produce the first three signs that come to mind. This helps make sure the signs are actually ones that the signer uses and knows. We paid participants for their time, and they consented to have their videos shared publicly for the purpose of developing sign language technology. We then went through and labeled each sign by matching it with a sign in the ASL-LEX database. This is a key innovation for two reasons. First, it means that we use ASL to label ASL, which results in much cleaner data. Second, because ASL-LEX has a wealth of linguistic information about each sign (e.g., its handshape, location and movement). That linguistic information, it turns out, is very useful. A sign recognition model that includes linguistic details about sign forms significantly improves how accurately the computer can recognize signs. The other neat thing about using ASL-LEX as the labelling system is that we can link this dataset to other datasets (e.g., ASL Citizen, ASL Play).
This post is based on "The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes" (Kezar et al., 2023, Proceedings of the ACM SIGACCESS Conference on Computers and Accessibility). The data is available at github.com/leekezar/SemLex.
