SignTag: A platform for collecting and annotating sign language data

High quality language data is the foundation of systems that recognize, generate, and translate human language. Technologies for sign languages have lagged behind those for spoken and written language, in large part because sign languages have historically been difficult to label due to the visual nature of the medium and lack of conventional written systems. Even the most well-documented sign languages are considered "low-resource" languages, and in these cases, data quality is critical. 

Why is there so little data?

The shortage of data arises for a few reasons. First, the format of sign language data is distinct. It requires video recordings of signers, not just text or audio, paired with structured labels that capture what is being signed, how it is signed, and by whom. Because sign languages do not have standardized written forms, labeling signs for the purposes of analysis presents theoretical and practical challenges. One solution is to scrape data from public sources, but these data are messy, may not have consent of the people in the videos, and often lack reliable labels. While that may be tolerable with sufficient data or for some applications, for other applications the quality of the data is too poor to be functional. 

How has the field historically approached data collection and labeling?

One approach to labeling sign language data  is to use written glosses (e.g., English translations), but this can prove challenging for lemmatization. Taking ASL as an example, an ASL sign may correspond to more than one English label (e.g., the same sign might be translated as CONNECT or BELONG depending on context) or no English label (e.g., the ASL sign commonly glossed as PEI-PEI, which describes someone exceptionally talented at something), and vice versa English labels may correspond to multiple ASL signs (e.g., the English word run corresponds to at least three ASL signs for running (jogging), running (operating a business), and running (a dripping nose)). More broadly, representing signs with written labels reproduces social ideologies that favor written/spoken language, a significant concern when deaf and hard of hearing communities routinely face linguistic discrimination and a lack of opportunities to learn and use sign language.  

Another solution is to use one of a number of notation systems that represent aspects of the phonological forms of signs, such as HamNoSys (Hanke, 2004) or SignWriting (Sutton, 1995). These systems label a sign by identifying things like its handshape, place of articulation, movement etc. However, such efforts are often inefficient and idiosyncratic, the same sign might be produced slightly differently each time, so in some cases a  different instances of the same sign receive a common label while and in other cases different signs with similar forms (homophones) are not distinguished from one another as distinct signs (Fenlon et al., 2015; Kezar et al., 2023). 

How does SignTag handle labeling?

A key innovation we propose here is labeling sign language data with sign language labels, rather than a printed intermediary. SignTag allows users to view a example video of a sign, and match it to an entry in a lexical database like ASL-LEX. That entry then serves as the label. The advantage of this is twofold: 1) it allows you to avoid the issue of using a written label to represent a sign, and 2) it means you can easily use all the linguistic information from ASL-LEX (e.g., information about the sign phonology, frequency, lexical class etc.).

How is SignTag different from other tools like Elan or SignStream?

Currently available tools for annotating sign language include ELAN (Crasborn & Sloetjes, 2008), a widely used multimedia annotation software, as well as sign language-specific options such as SignStream (Neidle et al., 2001) and iLex (Hanke et al., 2002). The main difference between these systems and SignTag is that SignTag is designed for both data collection and annotation, it is well suited to annotating short videos wehre the other systems are better for long videos of continuous signing, and it allows labelling signs with signs. Although it is possible in ELAN to link a sign language corpus as a lexicon service and use it as an external controlled vocabulary (Hochgesang, 2021), setup is not straightforward and requires users to download the full corpus locally.

How can I use SignTag?

Researchers using SignTag start by uploading a seed dataset made up of either videos or images. SignTag supports labeling with multiple fields at once which can be typical data field (e.g., free text, numeric, categorical, boolean flags), or they can be data types that are particularly useful for sign language data collection: video, and “sign lexicon.” The video type allows a user to submit a video (e.g., of themself signing) that then gets associated with the prompt. The system then allows users to annotate previously collected data (e.g., the videos) with any of the data types. The sign lexicon data type allows a user to select a label from a lexicon of signs like ASl-LEX (e.g., a  custom built database of uniquely identified signs with a corresponding video presenting the sign). Users could also label the videos with other data types (e.g., a field to represent handshapes, or lexical class). After the data has been labeled, the UI can be used to inspect the data visually, rendering each field based on the field type. The data can then be exported from the platform where each field is converted into a CSV representation and each video is also downloaded.