How UX Designers Can Prepare for a World with No Screens

by Jim Combs

Illustration by John LaPrime

They were in love and it was a dream vacation. Olivia and Milo had planned this trip since before they got engaged and now they were on their way.

As if by magic, the couple felt like the world’s best host had guided them to the most exquisite places on earth. People and places anticipated their arrival or reacted to any change of plan as the couple spoke, pointed, walked, looked, and proceeded normally along their itinerary. They were in heaven.

Behind the scenes, a holistically designed smart system pieced together every part of Olivia and Milo’s trip. Using an evolving range of bio and telemetric sensors, big data, and AI, this system collected both stored and sensed information about the couple in the most minute detail. Everything from their itinerary, travel preferences, and budget to the transportation availability, deals, and weather in their specific location was collected. The system even tracked what other travelers were doing around Olivia and Milo, as well as the couple’s in-the-moment actions and reactions on their travels to create their ideal vacation.

It’s a cliché, but the future is now
The connected experience described above is not that far off from reality. In fact, there are multitudes of experience-focused designers and engineers working to make this future real. We’re using advancements in gesture and voice technology to create entirely new customer experiences and shift how customers interact with brands.

Disney’s wearable MagicBand is a good example. This bracelet was launched in 2015 and it connects users to Disney’s theme park through a series of sensors. Anyone wearing the bracelet enters rides without waiting, “pays” for meals without getting out their wallets, and gets into their hotels without a key. It’s a human-computer interaction that removes the normal friction we typically have when we tap, click, or talk to a computer. In short, there’s no screen.

And Disney isn’t the only brand pushing into new ways of interacting with customers away from the screen. It’s spurring a ton of questions from a lot of the UX designers I know and work with who are starting to wonder whether their screen-oriented UX/UI skill set is going to translate to the growing field of screen-less user experiences. My answer is, “it depends.” The design research and strategic side of UX/UI remains intact. What’s changed is that we have a whole host of new, often invisible, interfaces to consider and make consistent across user journeys and touchpoints.

Here are three easy ways to grow your UX chops in this new world of evolving interfaces:

1. Bulk up on the experiential
We’ve all heard context is king when designing digital experiences. We research and explore the role of technology in the daily life of the user, and determine the influence of the user’s environment on our digital designs. Now we need to think beyond digital. Modern strategies call for a broad environmental context.

What’s on the screen is still important, but designing for the steps leading up to a screen or outside of a screen are equally important. A facial expression, wave of the hand, or physical gesture can initiate a task. Saying “Alexa” or “Hey Siri” turns on a new set of commands. Smart homes are using telemetry sensors to turn on lights when you walk into a room and turn them off when you leave. New cars unlock the door and start the car with your keys out of sight, and then will drive down the road without you having your hands on the wheel.

2. Pump up your design language
Design research should also incorporate and document observations of user behaviors outside of screen experiences. This includes an analysis and understanding of user motivations, challenges, and opportunities when a device isn’t present.

What’s the equivalent of a wireframe when one is describing a location-based interaction or one where the user travels from point A to point B? How does one specify how a camera sees individuals or objects in front of it and then reacts with appropriate actions to that input? How is the flow through a voice chatbot experience conveyed versus the flow through an app? Or how do you storyboard a robot’s thought process as it encounters humans, moves through buildings, and encounters obstacles?

While some screen design language remains the same when designing in this broader context, there is as much new or repurposed language in how we describe and document our designs. The fields of engineering, architecture, medicine and others have been tackling these issues for a while, so it pays to pay attention to their R&D.

3. Move your work to another dimension
If you primarily design screens and screen-based interactions, try layering on another dimension to your thinking. Follow your users around in physical locations and watch how they use your or your competition’s app, or any app for that matter.

What works, or more importantly, what doesn’t work when they’re in their car or as they move around their office? What happens when their hands are full? How would someone use your app if they couldn’t touch or see the screen?

Design your app to use the camera and the microphone to provide visual and voice recognition. Make use of location data and telemetric data to add nuance and additional information. Build an app that works in a fixed or movable location. Find additional data or services to integrate into your app to smooth out the user journey.

The move away from screens is happening fast
The move from desktop-first designed experiences to mobile-first experiences was fast. The same is true now for non-screen experiences. And while there is enough “transition-to-mobile” work to keep experience designers busy for the foreseeable future, that ship has been sailing for quite some time. Voice and AR/VR experiences are moving up many brands’ priority lists. Holodeck-like spaces and interactions already exist, and big data is becoming more and more cross-integrated.

Are your designs, products, services, and brands keeping pace?

And they lived happily ever after
They were in love and it was a dream vacation. Olivia and Milo had planned this trip since before they got engaged and now they were on their way.

Their playful Aibo woke them at 6:30am and a driverless ride swept them to the airport. As they arrived at the airport, an automated luggage cart met them curbside, scanning their intelli-suitcase for weight, contents, routing and destination, and transported the bag securely to the plane. The couple took an unfettered walk from curbside to their departure gate. Neither of them needed to touch a keyboard or swipe a touchscreen. And besides a 30-second wait to grab coffee, there were no lines to stand in, and the personnel who spoke to Olivia and Milo along the way greeted them by name, made them comfortable, and provided a high-touch quality to key points on their journey.

This scene was repeated once the couple arrived at their destination. There was seamless transport to the hotel and room check-in. Their vehicles, devices, and locations personalized trips to historical locations and museums with details only the smartest professor would know delivered in the style of the best storytellers. The entire restaurant, culture, and nightlife experience, shopping excursions, and leisurely strolls through architecture, parks, and gardens were expertly suggested and choreographed.

This was a trip they would remember for the rest of their lives.

Jim Combs

Related Thinking

Pick of the Bunch: T3 SXSW Panels


Pick of the Bunch: T3 SXSW Panels

It’s almost time for SXSW again! (Well, sort of.) But it is time for PanelPicker voting which will be open starting today through August 23 and we’re hoping to take the stage to discuss some important topics for us here at T3. Explore recaps of each PanelPicker submission below, and please vote for any topics that stand out to you.

Constantly Evolve. Then Evolve Again.


Constantly Evolve. Then Evolve Again.

When building a loyalty program, the foundation of a successful client partnership is built from the same core principles underlying all long-term partnerships

Intent Matters


Intent Matters

In discussions with clients who want to apply new or emerging technologies to their products or services, I often hear questions like, “What will we get?”