Gesture technology, invisible user interface and the evolution of human-to-machine interaction

Olof Schybergson
Bronwyn van der Merwe

Marketing Mag Australia:

The future of human-to-machine interaction will be a move away from touch screen technology to systems which anticipate natural movements and can be controlled by gestures, voice and ambient communication, according to global design and innovation consultancy Fjord.

Originating in 2001 with a focus on digital media, the company aimed to go beyond using digital for marketing communications, to focusing on services that people want and need in their lives.

In 2013, the company was acquired by Accenture and has since grown, so now Fjord within Accenture Interactive has expanded to 18 locations worldwide with a team of 700 people.

Today, Fjord works on systems which change the way people interact with technology, digital media and the world around them.

Marketing sat with Olof Schybergson, Fjord founder, and Bronwyn van der Merwe, Fjord Australia lead, to discuss the history of the company worldwide and in Australia, as well as the technology and the opportunities it presents.

Marketing: Please explain gesture technology and some of the other technology you’re working on.

Olof Schybergson

Olof Schybergson: What’s interesting about gesture technology actually goes beyond that specific technology, to a set of different ways that we interact with technology. Go back to human computer interface, engineering and design. For the longest period of time the way that people in general interacted with technology was essentially indirect interaction. Using a mouse to control a cursor on the screen and direct it that way. The smartphone really brought about a new wave of interacting with technology which is the touchscreen, that we now are all really accustomed to. Our kids, old people, anyone in between, are used to using touchscreens. That is now the standard, the dominant way that we interact with technology.

Alongside that dominant interaction paradigm, the shift from the PC or computer to the smartphone as the digital hub in your life has been very strong. To move from a bigger device, to something that you carry in your pocket. Those two things have changed how we interact with computing.

Now we see the evolution of gesture based technology, voice recognition, artificial intelligence, ambient interactions and computing and signalling. A whole range of new ways of input and output between technology and people that is actually starting to add to and make the interaction that we have with services, with brands, with technology, potentially more natural for people, but also breaks that very strong dominance of the touchscreen as the only real way of interacting with technology.

In very normal interactions day-to-day, we’re already using gesture based technology, and then we’re also starting to see it integrated more into digital experiences, one kind or another. We’re also very excited about voice technology, voice recognition and voice interaction. That’s a very human way of interacting. Our voice is a big part of our personality, how we communicate, how we talk to people, and increasingly the technology is very natural. It’s much more natural actually than touch screens or other form of typical interaction with technology. We see the emergence of our voice in lots of different places.

M: What are some of the potential setbacks that the new technology has been facing in terms of integrating into mainstream use so far?

OS: As you bridge towards more natural forms of interaction, for example, voice recognition and technology talking to you. What’s interesting about that is that as humans, we start to apply the same expectations as we do with human communication. If the content is right, but it’s said at the wrong time, it feels a bit too intrusive too quickly. If it’s using the wrong tone of voice, all of these things are small nuances that we pick up on in our human-to-human interaction very quickly and very easily. And now we start to use the same filters with technology. It means that it’s really challenging actually, to get all of those things right, at all times, with technology. That’s one of the inherent challenges within it.


Bronwyn van der Merwe: An interesting challenge with gesture, is that there are cultural significances for given gestures. There isn’t necessarily a universal gesture that works with every culture. Part of what we’re doing at Fjord is exploring that space and thinking about what other gestures that we can use, and where do they need to be different for different cultural situations to control the world around us.

M: Gesture technology and invisible user interface will change the way people interact with technology, can you think of any ways that it may change ways people interact with each other?

BvdM: One of the key things that invisible UI or natural interfaces change our lives is that they stop the dominance of the screen in front of your face. In a family environment, for example, with Amazon Echo, which is a voice operated device that allows you to turn music on and off, ask what the weather is, find out all sorts of information. Within a family environment, that immediately changes the dynamic, because you no longer have to stop, look at a screen, you can interact with your children and you can make voice request to Echo and it’ll give you that answer. It changes the dynamic in an environment where the screen is less dominant.

OS: That’s a really important point. The social dynamic with a screen tends to be personal. You cocoon into your own space, and that might actually pull people apart rather than pushing them together. We have a colleague at Accenture who bricked Amazon Echo, this little device, into his home, and told his kids, ‘let’s use that for getting information from the internet, let’s not use our screens at home’, and it completely changed the dynamic in the household for the better.

As digital is becoming more integral in our lives across everything, whether it’s shopping, healthcare, exercise, insurance, banking, finances, across everything that we do, digital is becoming really central. As that happens, people are also recognising that they crave for deep human interaction and experiences. You have this counter trend that we call digital dieting which is finding the moments when you disconnect from screens, finding the moments where you put technology to the side or use technology only to enhance the face-to-face interaction rather than allow it to disrupt.

Read the full interview on Marketing Mag Australia


Olof Schybergson
Bronwyn van der Merwe

More Stories from Fjord