BANNER SMALL.png
 

How might we use object detection to help people living with visual disabilities?

Thiea Vision is a start-up that aims to help people living with visual impairments navigate the world. It is a phone app that uses a neural network to process a video feed from a wi-fi enabled camera and convert it into audio feedback.

 
 
 

the problem

Living with a visual disability can be tough. This is especially true for people living with severe visual impairment as all of our built environments do not cater to somebody with poor vision. Everything from wayfinding to navigating roads to something as simple as finding a coffee mug can be really challenges for someone with visual disabilities.

 
 
RESEARCH SUMMARY-02.png

With the youngest of the baby boomers hitting 65 by 2029, the number of people with visual impairment or blindness in the United States is expected to double to more than 8 million by 2050

RESEARCH SUMMARY-03.png

As of no, people living with visual impariments have to almost “hack” their way around life as society as a whole relies primarily on vision for communicating. Everything from way-finding to identifying the correct buttons on a microwave are complicated by vision loss.

 

the technology

 
Gif_tech.gif
 
 

The idea was to use a neural network that runs real time object detection to identify objects and convert visual into audio feedback for people living with visual disabilities. The idea started out over a cup of coffee with a friend who was working on TensorFlow.

 
 

the timeline

 
timeline-01.png
 

 

RESEARCH

RESEARCH SUMMARY

RESEARCH SUMMARY.png
 
 
 

what i read

23.7 MILLION PEOPLE ARE VISUALLY IMPAIRED IN THE UNITED STATES ALONE

23.7 MILLION PEOPLE ARE VISUALLY IMPAIRED IN THE UNITED STATES ALONE

OUT OF THE 23.7M, 10 MILLION ARE CONSIDERED SEVERELY VISUALLY IMPAIRED

OUT OF THE 23.7M, 10 MILLION ARE CONSIDERED SEVERELY VISUALLY IMPAIRED

1.3 MILLION PEOPLE ARE LEGALLY BLIND. BEST CORRECTED VISION OF 20/200

1.3 MILLION PEOPLE ARE LEGALLY BLIND. BEST CORRECTED VISION OF 20/200

THE SINGLE MOST COMMON CAUSE AT 43% OF ALL CASES OF VISUAL IMPAIRMENT

THE SINGLE MOST COMMON CAUSE AT 43% OF ALL CASES OF VISUAL IMPAIRMENT

CATARACTS AND GLAUCOMA ARE THE 2ND AND 3RD BIGGEST CAUSES OF VISUAL IMPAIRMENT AT 33% AND 2% RESPECTIVELY

CATARACTS AND GLAUCOMA ARE THE 2ND AND 3RD BIGGEST CAUSES OF VISUAL IMPAIRMENT AT 33% AND 2% RESPECTIVELY

 

What I heard

As far as navigation is concerned, the biggest obstacle is fear
My brother in law put my biece’s play-pen in the middle of the room and forgot to tell me about it. I tripped on it and hurt myself pretty bad
Overhanging objects that that cannot be detected from the ground with a white cane such as trees and low hanging objects are the the most scary
The primary predator of the blind person is an automobile
 

what i felt

I spent a weekend with a blindfold on to gain some empathy for what it feels like to live with a visual disability. This led to some interesting insights.

 
IMG_6564.jpg
Shampoo and conditioner bottles are exactly the same. This is true for so many products and buttons.

Shampoo and conditioner bottles are exactly the same. This is true for so many products and buttons.

While Braille sign-boards exist, there is no standardization of where they are placed.

While Braille sign-boards exist, there is no standardization of where they are placed.

 
All bills in the US are exactly the same size. This makes differentiating $100 from $1 difficult.

All bills in the US are exactly the same size. This makes differentiating $100 from $1 difficult.

Tactile pavements exist but not everywhere. It is difficult to know where to feel for them and where not.

Tactile pavements exist but not everywhere. It is difficult to know where to feel for them and where not.

WHAT PEOPLE SAID

 

We sent out surveys asking people living with visual disabilities about some of the challenges they felt and what their biggest problems were. We also asked them how they navigated these challenges and what their own personal “hacks” were. (n=21)

 
 
3.jpg
 
 

 
 

synthesis

stakeholder map

I charted out the various stakeholders involved in the ecosystem that surrounds a person living with severe visual disabilities to understand the different intervention points that come into play.

 
 
 
 
 

empathy map

Instead of creating user personas and journey maps, I decided to create an empathy map for my target audience. This helped me verbalize the biggest pain and gain points and effectively communicate my findings. I felt that a journey map or a persona would be too limiting as the target audience is very broad in terms of likes, interests and other personality traits. The only thing they have in common is their visual disability.

 
 

THE IDEA

 
andrik-langfield-petrides-328825(1).jpg

WHY

Total loss of vision in both eyes is considered to be 100% visual impairment and 85% impairment of the whole person. It is the 21st century and technology has finally reached the point where we can make the world more accessible for everyone.

image(5).png

HOW

By creating a camera paired with a smartphone to run real time object detection on a live video feed. The smartphone processes the video and provides audio feedback to the user.

USER FLOW

ServiceFlowTheiaVision-01.png
 

The biggest challenge with this project was to translate raw data (bounding boxes and probability percentages) from the neural network into a user-friendly format. Since the primary sense for someone with a visual disability is their hearing, the audio feedback must be extremely selective. Based on user feed-back, 5 modes were chosen: Navigation, Object Detection, Currency Identification, Object detection, and Text to Speech.

 

Version 1

SCREENS

Screens.jpg

CHECKING ACCESSIBILITY FOR COLOR BLINDNESS

 
Screenshot 2019-05-25 at 3.43.04 PM.png
Screenshot 2019-05-25 at 3.43.28 PM.png
 
 

RATIONALE BEHIND SCREENS

I consciously decided to create as simple an interface as possible. Most users interact with apps on their phones by using the talk-back feature on their phones. However, a small subset of users that have severe visual disabilities are still able to see objects up close. To meet their needs, the screens have big bold text with high-contrast. All of the different modes also have different colors to make it easier to differentiate between screens.

 
 

VERSION 2

THE SCREENS

 
Perspective+App+Screens+Mock-Up.jpg

CHECKING ACCESSIBILITY FOR COLOR BLINDNESS

 
Protanopia

Protanopia

Deuteranopia

Deuteranopia

Protanomaly

Protanomaly

Deuteranomaly

Deuteranomaly

 
 

Based on user-feedback, I modified the screens to include tabs to indicate the current mode. I also updated the Visual Design to simplify screens and get rid of unnecessary elements.

SKETCH LINK

 
 

PRODUCT

DEVELOPMENT

 
 
 
2.gif
 

Form Development

FormDevelopment-03-03.png

MOCK UPS

12.jpg

FINAL RENDER

Camera.png
Docking.png
Person.png
 

The moment we decided to design a product that was mounted on to glasses, we knew it would be impossible to make the product invisible. We decided to make the product a fashion accessory instead to make it something that users would be proud to wear.

 
Movement.gif
Hero.jpg
1-Animated.gif

Design

The app was designed to be used by people with visual disabilities. To make it as easy as possible, minimal and large text, along with a range of bright colors for each screen were used.

Micro-Animations.gif

Micro-Animations

Micro-Animations were designed with the intent of optimizing the user experience for people with low vision.