Apple WWDC 2019 Scholarship winning project.
Music Hands is an interactive Swift playground that teaches American Sign Language (ASL) fundamentals with Apple’s Vision framework! Play along by following the highlighted bubbles at the top of the screen and gesture letters A through G in sign language to play unique compositions of chords developed in Logic Pro X. At the bottom of the screen, you’ll see hand gesture icons appear as the melody progresses to indicate which gesture to make next. Each frame streamed through the front camera is carefully processed by a custom model. After discovering there were a limited amount of datasets for the project, images of 50 unique hands for each of the seven letter gestures were captured and labeled from ASL Alphabet tutorials on YouTube manually, using the screenshot tool on MacOS. Core ML Create was then used to develop a 115KB model for classifying hand shapes. As someone who is new to the machine learning space, the Create ML UI was a refreshing and playful place to experiment with classifying models with a minimal amount of code overhead involved.
The original vision of the project was to create a “Guitar Hero-like” experience for learning American Sign Language. Picking up a new language takes time and plenty of repetition. In a fun gamified experience, this repetitive process of learning has the potential to go by much more quickly, especially when music is involved. This has only recently been possible with the advancements of rapid image recognition and the accessibility of machine learning tools, with Create ML and Turi Create, for the masses. Just a few years ago, the task of sign language classification could have easily been seen as cutting edge research. Today this project can be put together within a week. While the first 7 letters of the alphabet mapped well to the 7 major keys of a musical scale, future versions of the project could potentially support all 26 characters of the alphabet. Multiple languages and symbol support could also be added to the app to encourage an even more meaningful exploration of sign languages around the world.
