Hi Karen, thanks for the comment! We sure hope that SAIL can jumpstart a transformation of ASL learning tools. 

We haven't had any new signers test SAIL yet, because right now we are transforming the motion capture recordings into avatar characters. Until those are created, the motion capture recordings are not user-friendly--they look like dots moving on a screen (aka, point-light displays). I am really curious to hear what our first sign-naive users think--their feedback will be really valuable to future iterations of the project.

The Motion Capture data records native signers producing basic ASL content, much the same way as Hollywood studios use motion capture to record characters movements and facial expressions for use in movies. The 16 cameras capture the hand, body, finger, and face movements that are critical to ASL with extremely high accuracy. We then use those recordings to render our signing avatars. The other type of motion recording which our project uses was not really highlighted in the video, although there were glimpses of it. This method is gesture tracking, in which a LEAP gesture tracker will track the SAIL user's own movements as they interact with signing avatars, allowing them to see their own sign productions alongside those of the signing avatars. 

Technologically, this is absolutely fascinating!  Your video is well done.  Good pace.  Just the right amount of information.  I have been working with students and administration at the Wisconsin School for the Deaf for a number of years to address language access issues for deaf students in astronomy.  Outside some finger-spelling and a few basics, I can confidently sign "Finished? Save to desktop" So. . . I am wondering how you imagine your signing avatar helping me prep for class, assuming I am working with a teacher who signs.

Hi Kate, thanks for watching and I am happy to hear you enjoyed the video!

We envision that in the not too distant future, you could take brief ASL lessons through your personal VR device, like an oculus rift or even a mobile-phone based VR setup. So maybe after dinner, you'd put on your VR headset and do a few of the gamified ASL lessons. In this future iteration of SAIL, the lessons might involve points, levels, and feedback about your signing. So you could complete ASL I in VR, interacting with avatars who are demonstrating high-quality ASL content. That is our dream for SAIL!

Thanks for your interest! I am happy to hear you enjoyed the video! Great question--there are certainly a lot of challenges in creating useable VR content. So far, some of those challenges in regards to our project involve the high fidelity that we require in order to show people fluid, fluent ASL. We need very high definition on the face, arms, and every joint of the fingers, to ensure that the eventual signing avatars can produce beautiful, natural ASL. Another challenge is in getting all the different components of our system connected to one another. We have oculus VR goggles, a LEAP gesture tracker, working on a Unity-based gaming engine. There are certainly challenges with making sure every element is working in sync. Finally, we work really hard to make sure our avatars and VR environment feel natural and pleasant. It is easy to feel creeped-out or unsettled when entering a virtual environment, and we do not want to be trying to teach people when they're feeling ill-at-ease. So making a smooth, natural experience is a top priority of ours. Let me know if you have any other questions. I am excited to learn more about your work!

Hello Sarah! I'm loving your enthusiasm!

The fingerspelling and counting were actually just demos that we used for this video :-) In this version of SAIL we're working on now, the ASL teacher will teach the user about 30 signs--things like BACON, EGGS, MILK while standing in a kitchen environment, and SWING, PLAY, JUGGLE, while doing a games-related activity. We have structured 4 brief ASL lessons, each of which contains a few target ASL signs that are loosely grouped by theme. 

Our avatars currently produce ASL which has been pre-recorded from native signers in Motion Capture suits, but in the future, we hope to use the avatars to piece together signs and produce new content, sort of like you're describing. In the further out future, an avatar could use machine learning to draw upon their knowledge base and produce new content in an adaptive fashion, sort of like how Alexa and Siri can respond creatively to questions using their knowledge bases. Hope that helps you envision the future of our project! It sure is wild to think of what this type of technology may be able to do someday, in the hopefully-not-too-distant future! 

Oh my gosh you must be so excited to get up and go to work every morning!! :-D It IS wild to envision, I agree - almost as good as in the 90s envisioning this type of conversation on the new Internet haha - appreciate all the time you are taking responding to our questions and sharing more about the work. Congratulations!

I am so glad you enjoyed the video! The future of learning is indeed and exciting thing to envision! 

Thanks, Amy! Signing avatars hold so much potential to increase the inclusivity and accessibility of these new technologies. I will make sure to check out your video.

I am glad you enjoyed it!

Thanks so much!

Thanks so much, Diana! At this time, SAIL is being developed and we are working towards a proof of concept--showing that all these systems can work together and provide a learning experience for users. Once we have created the proof of concept, we certainly hope to continue to improve and develop SAIL, and in the long term, we would love to share the SAIL system with schools, individual users, and the Deaf and ASL-learning communities. Maybe in the future people can download SAIL onto their personal VR systems at home!