Latency is the new 5G speed battleground which will enable XR and AR

Ian Fogg, VP Analysis

For many kinds of immersive interactive media, mobile download speed is not the only important measure of speed. For mixed or extended reality (XR), augmented reality (AR), 360-degree videos, and virtual reality (VR) the latency of a mobile network is even more important than speed because it enables the experience to feel immediate, real and responsive. While latency has improved in recent years on 4G mobile networks, the designers of 5G — which is launching in 2019 for smartphone users — aim to deliver a disruptive improvement in mobile latency.

Smartphone users on current U.S. 4G networks experience average real world latencies between 60.5ms and 42.2ms across 40 US cities. While Austin may be a media and technology center which hosts the South by Southwest (SXSW) Conference & Festivals each year, its latency ranks just 32nd out of 40 cities analyzed by Opensignal. Busy New Yorkers have a highly responsive mobile network experience — at least by today’s 4G standards — which matches the stereotype of life in the Big Apple.

However, the 5G technology arriving for U.S. smartphone users in the next few months aims to reduce the latency of the radio network to under 1ms. And, when 5G core networks arrive, they too should dramatically improve latency on current 4G networks and enable greatly improved immersive experiences such as 360-degree video streaming and mobile AR.

360 VR video needs low latencies to compensate for limited 4G network speeds

Why should 360-degree video benefit from low latencies? Isn’t fast download speed enough? As Opensignal has shown, video experience is not the same as download speed. And, 360-degree video is especially challenging because of the sheer volume of data that needs to be transmitted to represent the complete 360-degree image. How 360-degree video streaming works today is to adapt the streaming quality of different parts of the 360-degree image, based on where the user is looking, because current 4G mobile networks are not consistently fast enough to stream the whole 360-degree video at full detail and compression level. The part of the 360-degree image where a viewer is not looking is transmitted at a lower quality than the part the user is watching, and the degree of video compression applied changes dynamically based on where the viewer is choosing to look.

When a viewer turns their head around, the video responds by increasing the quality of that part of the video. If the latency is poor, when a viewer moves their head, the video quality will take much longer to react and the viewer will briefly be forced to watch a low quality video stream until the network catches up and the stream improves and comes into focus. This experience is deeply irritating — sometimes nauseating — because of the lag experienced when the video renders at low quality, then improves.

Latency is in effect the responsiveness of a mobile network. One way to view it is to think of the reaction time between seeing a baseball fly through the air and being able to accurately judge when to strike the ball. This very experience was one of the 5G latency demos shown recently at Mobile World Congress (MWC) by an Asian operator, which was delivered entirely in VR.

VR baseball

Smartglass AR requires 5G’s responsiveness

AR will be even more demanding on mobile network latency. While it’s just about possible to stream 360-degree video today, a rich AR experience on future mobile devices such as smartglasses will require a very different mobile network experience that only 5G will be able to support. Future AR experiences will be completely different from the constrained pocket-sized AR apps of pre-2019. Currently, most AR games and apps create the experience on the smartphone alone to side-step the lack of mobile network quality.

AR smartglasses proponents foresee a world where a user has an instant flow of information about the objects in the real world they look at. But this database of information will be too large and far too dynamic to be stored locally on a device. Instead, smartglasses will need to communicate continuously with the cloud in real time to ensure the wearer has access to this vast digital knowledge base at all times. Wherever the wearer looks they will expect instant information about whatever comes into view, whether it’s the opening times of a shop, when the next train will arrive or where they can find the nearest electric vehicle car-charging parking spot.

But we will need to test the real-world experience of the 5G networks that launch this year to make sure that mobile operators actually deliver on their promise of single-digit latency. For the static XR experiences of today to transform the world they will have to become mobile and available 24x7, just as the smartphone took PC websites and made them ubiquitous to everyone, everywhere and all of the time.

The AR, VR and XR demos that were on display at MWC and the demos we're sure to see at SXSW will no doubt be impressive. But these experiences will never turn into real mobile applications and services if 5G doesn't truly make networks more nimble and significantly more responsive.