The AI race toward inclusion
A product innovation use case of designing for inclusion
1. Building empathy
I’ve started learning ASL (American Sign Language) at the beginning of this Fall — you can read more about my insights on how ASL can make you a more inclusive innovator here.
As a designer (an a very empathic human being), part of my interest in taking ASL was to build empathy with the deaf community. I travel a lot, and often alone; the most “difficult” travel experiences have been traveling to destinations where I didn’t speak the language, and where the local population didn’t speak English or French (the two languages I’m most comfortable communicating in).
2. Gathering insights
Since starting the ASL course, I’ve often wondered what interactions for a deaf person is like with public services: asking for help in a store; asking the waiter/waitress to explain the menu; or even simply asking for directions. I’m sure some people would respond that “with a mobile phones and google a few clicks away, it’s not that bad”. The truth is they are still ripped off of the simple convenience of just asking, and well… phone batteries don’t last forever!
As COVID hit, I also realized how important body language and facial expressions are important in our communication. The team I worked with didn’t really want to turn on their cameras, so we were left to communicate simply with words and tone of voice. I often wondered: how would this team collaborate and work effectively if we had a deaf colleague? And sadly, the answers I could think of didn’t sound too good. The blatantly clear “answer” to this was the fact that we probably wouldn’t hire a deaf team member, because we would have communication issues, and it team productivity would likely go down.
Part of the communication issue can thankfully be solved thanks due globalization. With the increasing number of global teams, tools like Microsoft Teams have come up with a live caption feature, and the option to translate that live caption in the user’s preferred language; this could be one solution.
3. Defining the problem
Fast forward to today: I was watching the recording of a webinar demo’ing a tool for remote collaboration when I had an aha moment. The presenter on the webinar mentioned “If you cannot hear us, click on the settings to turn on your audio.” I thought, what if audio isn’t the problem? What if I just cannot hear?
4. Prototype ideation
A solution came to mind: what if you could have a plugin that would automatically ‘listen’ to the audio coming out of your screen, and have that audio translated back not as a live caption (it can get exhausting to read), but as an Avatar that could translate that speech into a person signing the speech?
In essence, like most innovations, the solution is inspired by bits and pieces of other existing solutions:
- A plugin, to allow the translation feature to work on any website (i.e. accessibility anytime the user is online)
- Voice to text: capturing what is said into text, so the deaf person can understand what is being said (inclusive step #1).
3. As opposed to reading the text caption, the user can choose to ‘listen’ in a language format that is more natural to them: sign language. The caption text is thus translated into sign conversation via an avatar. This part of the solution is actually something that the Enterprise Europe Network is working on! An amazing use of AI in graphics, and a great illustration of how designing for inclusion fuels very interesting tech developments — I can totally see this being used to create motion pictures, hello Pixar Studio!
Unique challenges
As I started thinking about the solution more and more, I realized that the grammar and structure we speak in (speech) is very different from the one used in ASL (signing). For instance:
[Speech] Do you like playing with your dog?
[ASL] You like playing with your dog, you?
Another complexity is the use of motion (signing) driven by AI. Similarly to ‘speech to text’, the algorithm needs to understand what is being said to find the suitable translation. In addition, the translation needs to be converted from text to an avatar, with details with respect to:
- Facial expressions (frowns and raised eyebrows)
- Head motion (head nod, head shake, tilted head)
- Arm and hand motion (sometimes symmetrical, sometimes asymetrical)
- Precise finger motion and positioning
- And a combination of any of the above!
Even though we might have figured out how to handle differences in grammar and the order of words in spoken languages, thanks to large volumes of data available to train, I don’t believe there is a database that captures sign language in written form anywhere — if there is, please let me know in the comments!
How can we ensure such data is included as we develop machine learning algorithms that will define our future and guide our future behaviours? Exclusion today as algorithms are being trained means exponential exclusion tomorrow. Now is the time to be conscientious in defining and inclusive tech-enabled future.
In addition, given the very regional use of sign language, this could also be a solution used for various deaf communities to communicate with one another.
Inclusive design is ripe for experimentation and innovation!
And remember that designing to include one community often extends the benefits to a larger group of individuals.
5. Call to action: Prototyping & Testing
If you are a developer or know a developer looking to work on a pro-bono project around designing machine learning solutions around inclusion, please contact me to collaborate on this!
Feel free to check-out my portfolio and/or reach out to grab a tea (or coffee) with me — if you know me well though, I’m a tea lover!
