Malaysia case study: Why fast download speed is not equal to good video experience

Hardik Khatri, Technical Analyst

It's very common to think that fast download speeds translate to a good mobile video experience, but that’s certainly not the case when we take a deep dive into real-world data. Speed certainly is an indicator of mobile Video Experience but not the only factor to determine the quality of video we see. In this analysis, Opensignal investigated our smartphone users’ mobile network experience on Malaysia’s five major networks — Celcom, Digi, Maxis, U Mobile, and Unifi — to compare their 4G Download Speed and Video Experience (measured on active video testsside by side, as well as delve further into users’ standard definition (SD) and high definition (HD) Video Experience. This analysis covers three 90-day aggregation periods ending on 30 November 2018, 28 February 2019, and 31 May 2019, which are denoted by Period 1, Period 2, and Period 3 in the charts throughout respectively.

Comparing 4G Download Speed Experience and Video Experience, we noticed that across all three periods our users on Maxis’ network enjoyed the fastest average 4G Download Speeds of at least 20 Mbps, while the operator garnered a Very Good rating on 4G Video Experience (65-75 points on a scale of 100). On the other hand, our users on Celcom’s network experienced average 4G Download Speeds lower than 20 Mbps — 16.1 Mbps, 17.2 Mbps, and 19.2 Mbps in Period 1, 2 and 3 respectively — but the operator still managed to achieve a Very Good video rating in two out of three periods, despite having a much greater difference of at least 3.9 Mbps in its users’ download speeds. We also observed a similar scenario with two other operators — U Mobile and Unifi — during Period 3, where our users on U Mobile’s network experienced average 4G Download Speed of 6.6 Mbps and the operator obtained a Poor Video Experience rating (0-40 points) with a score of 38 points, whereas our users on Unifi’s network experienced an average 4G Download Speed of 7 Mbps but the operator earned a Fair Video rating (40-55 points) with a score of 52 points. These scenarios clearly highlight the discord that exists between the two metrics: Download Speed is not a good indicator of Video Experience.

Opensignal also examined if there was any correlation in the percentage change of these two metrics and noticed that an increase or decrease in the 4G Download Speed experienced by our users did not affect their Video Experience with the same magnitude on the above-mentioned networks. 

This was more evident on some of the networks like U Mobile, DiGi, and Unifi. We saw that between period Period 1 and Period 2 our users on U Mobile’s network experienced a 40% decrease in average 4G Download Speed, but their Video Experience only fell by 5%, and between period Period 2 and Period 3 our users experienced an increase in average 4G Download Speed by 23% yet their Video Experience increased by just 4.2%. We saw a similar difference in the metrics on DiGi and Unifi’s network, where our users on DiGi’s network experienced a decrease of average 4G Download Speed by 2.4% but their Video Experience score increased by 2.2%, while on Unifi’s network our users experienced an increase of 4G Download Speed by 4.2% but their Video Experience score increased by 18.7%.

Video Experience can also be heavily impacted by operators’ policies since globally many operators use network optimization technologies to restrict the level of video resolution their customers can access on their phones. As Opensignal tests video streaming at different resolutions, any downgrading of video quality — say, from HD to SD — would have an impact on our Video Experience scores. Hence we further looked into the HD and SD Video Experience on 4G connections individually.

With users experiencing speeds above 15 Mbps in all three periods on Celcom and Maxis’ networks, we saw both operators achieve a Very Good rating for Video Experience on SD resolution compared to HD video resolution, where only Maxis was able to achieve a Very Good rating across all periods — although Celcom was not far behind with Very Good ratings on HD video resolution in Period 2 and Period 3. We noticed a more eye-opening scenario on U Mobile and Unifi’s networks, where our users experienced average 4G Download Speed from a range of 5 Mbps up to 9 Mbps in all three periods. Both operators’ users, in this case, were able to experience a Good rating for Video Experience on SD video resolution but only a Poor rating on HD. We also noticed a sharp rise in the HD Video Experience score on Unifi’s network during period P3, which may be a result of the removal of a network management policy. 

Download speed has been the focal point of the mobile industry ever since the first data networks came online nearly two decades ago. However, speed is just an abstract measurement, not an indication of how consumers actually experience the internet or applications on their mobile phones. Speed certainly has a bearing on mobile Video Experience, but as this analysis shows, it is by far not the only factor determining the quality of video users see.

At Opensignal we have opted for a forward-looking approach with a direct test of Video Experience which includes measuring of load-time to the first frame and playback stalling occurrences. Video streaming is the largest category of traffic  carried on mobile networks in Malaysia, and this consumption will grow dramatically to keep pace with consumer demand and a growing user base.

It's time for the regulators and operators in Malaysia to turn their attention away from just testing speed and instead focus on all of the factors that go into ensuring a good mobile network experience. Video Experience is an ideal place to start as it depends on a wide range of elements: the availability of 4G connections, the latency of the network and the consistency of connections, as well as the individual data management policies of operators. To measure consumers’ Video Experience, Opensignal’s analysis demonstrates it’s necessary to measure video quality directly and not estimating it based on download speed tests.

Opensignal's Video Experience methodology
Opensignal's Video Experience metric is derived from an International Telecommunication Union (ITU)-based approach for measuring video quality. Our tests sample video at multiple resolutions accessed from multiple content providers, and they weigh three main criteria: the load time before the video begins playing, the stalling rate characterized by stops and stutters in the video playback, and the level of picture resolution. We measure Video Experience on a scale from 0 to 100 — the higher the score, the better the Video Experience. Furthermore, we divide those scores into ranges to determine their rating. A score that falls within 75-100 is Excellent, 65-75 is Very Good, 55-65 is Good, 40-55 is Fair and 0-40 is poor. An Excellent score is a big achievement indeed, meaning fast load times and practically non-existent stalling at all resolutions. As we descend down the ratings, loading times get longer and we encounter more stops and stutters in the video stream. At our worst rating, Poor, users don't have a good experience at all with very long waits for the video to start and frequent stalling even at the lowest resolutions.