In this blog post, we’re taking a deep dive into the very core of Opensignal: explaining the metrics we use, what they all mean and what their roles are in measuring the real-world mobile network experience as users see it.
How do we collect the data in the first place?
We collect billions of individual measurements every day from over 100 million devices worldwide. We collect data every day of the week, at all hours and in all the places people live, work and travel: no simulations, no predictions, no idealized testing conditions. Our data comes from actual smartphone users and we report users’ actual network experience, whether they are indoors or out, bustling in a busy city or trekking in the countryside.
We collect the vast majority of our data via automated tests that run in the background, enabling us to report on users’ real-world mobile experience at the largest scale and frequency in the industry. These automated tests are run at random points in time and therefore represent the typical experience available to a user at any given moment.
Video Experience – Measuring real-world mobile video streams
Opensignal’s Video Experience metric quantifies the quality of video streamed to mobile devices by measuring real-world video streams over 3G and 4G networks. The metric is derived on an International Telecommunication Union (ITU)-based approach, built upon detailed studies which have derived a relationship between technical parameters, including picture quality, video loading time and stall rate, with the perceived video experience as reported by real people. To calculate video experience, we are directly measuring video streams from end-user devices and using this ITU approach to quantify the overall video experience for each operator on a scale from 0 to 100. The videos tested include a mixture of resolutions and are streamed directly from the world’s largest video content providers.
The following scale can be used to relate the Video Experience scores to the actual experience users receive:
- 75 or above Excellent: Very consistent experience across all users, video streaming providers and resolutions tested, with fast loading times and almost non-existent stalling.
- 65 < 75 Very Good: Generally fast loading times and only occasional stalling but the experience might be somewhat inconsistent across users and/or video providers/resolutions.
- 55 < 65 Good: Less consistent experience, even from the same video streaming provider and particularly for higher resolutions, with noticeably slower loading times and stalling not being uncommon.
- 40 < 55 Fair: Not a good experience either for higher resolution videos (very slow loading times and prolonged stalling) or for some video streaming providers. Experience on lower resolution videos from some providers might be sufficient though.
- Under 40 Poor: Not a good experience even for lower resolution videos across all providers. Very slow loading times and frequent stalling is common.
With video being the single largest category of traffic carried on mobile networks and consumption expected to grow to keep pace with consumer demand, this is an extremely relevant metric. It is another key element in Opensignal’s ongoing mission to measure the actual consumer experience on mobile networks.
In addition to Video Experience, we report on two supporting metrics related to video:
- 4G Video Experience: The average Video Experience score across an operator’s 4G/LTE connections
- 3G Video Experience: The average Video Experience score across an operator’s 3G connections (e.g. UMTS/HSPA or CDMA 1X EV-DO)
Opensignal’s Games Experience metric is a measure of how mobile users experience real-time multiplayer mobile gaming on an operator’s network. Measured on a scale of 0-100, it analyzes how the multiplayer mobile Games Experience is affected by mobile network conditions including latency, packet loss and jitter to determine the impact on gameplay and the overall multiplayer Games Experience.
The Games Experience metric quantifies the experience when playing real-time multiplayer mobile games on mobile devices connected to servers located around the world. The approach is built on several years of research quantifying the relationship between technical network parameters and the gaming experience as reported by real mobile users. These parameters include latency (round trip time), jitter (variability of latency) and packet loss (the proportion of data packets that never reach their destination). Additionally, it considers multiple genres of multiplayer mobile games to measure the average sensitivity to network conditions. The games tested include some of the most popular real-time multiplayer mobile games (such as Fortnite, Pro Evolution Soccer and Arena of Valor) played around the world.
Calculating Games Experience starts with measuring the end-to-end experience from users’ devices to internet end-points that host real games. The score is then measured on a scale from 0 to 100.
The Opensignal mobile Games Experience has the following categories:
- 85<100 Excellent The vast majority of users deemed this network experience acceptable. Nearly all respondents felt like they had control over the game and they received immediate feedback on their actions. There was not a noticeable delay in almost all cases.
- 75<85 Good Most users deemed the experience acceptable. Gameplay experience is generally controllable and the user receives immediate feedback between their actions and the outcomes in the game. Most users did not experience a delay between their actions and the game.
- 65<75 Fair Users found the experience ‘average’. In most cases the game is responsive to the actions of the player with most users reporting that they felt like they had control over the game. The majority of players reported that they noticed a delay between their actions and the outcomes in the game.
- 40<65 Poor Most users found this level of experience unacceptable. The majority of users reported seeing a delay in the gameplay experience and they did not receive immediate feedback on their actions. Many users felt a lack of controllability in the Games Experience.
- 0<40 Very Poor Nearly all users found this level of experience unacceptable. Almost all users experienced a noticeable delay within the game, with most of them not feeling like they had control of the gameplay. The vast majority of players didn’t receive immediate feedback on their actions.
Voice App Experience – Quantifying voice quality
Opensignal's Voice App Experience measures the quality of experience for over-the-top (OTT) voice services — mobile voice apps such as WhatsApp, Skype, Facebook Messenger etc. — using a model derived from the International Telecommunication Union (ITU)-based approach for quantifying overall voice call quality and a series of calibrated technical parameters. This model characterizes the exact relationship between the technical measurements and perceived call quality. Voice App Experience for each operator is calculated on a scale from 0 to 100.
- 95 or above: Excellent: Most users are very satisfied. Operator provides consistently good OTT voice quality experience across the customer base.
- 87 < 95: Very Good: Most users are satisfied. Operator generally provides good OTT voice quality experience. Occasionally, there may be some impairments to the call, primarily related to level of loudness.
- 80 < 87 Good: Many users are satisfied. Minor quality impairments experienced by some users. Sometimes the background is not quite clear, it can be either hazy or not loud enough. Clicking sounds or distortion is very occasionally present.
- 74 < 80: Acceptable: Users are satisfied. Perceptible call quality impairments experienced by some users. Short duration of clicking sounds or distortion can be heard, and/or the volume may not be sufficiently loud. Listener is generally able to comprehend without repetition.
- 66 < 74: Poor: Many users dissatisfied. Call quality impairments experienced by many users. Distortion, clicking sounds or silence experienced during the call, which is perceptible and can be annoying.
- 60 < 66: Very Poor: Most users dissatisfied. Significant call quality impairments experienced by most users. Occasional instances of distortion, clicking sounds or silence experienced during the call. It can be difficult to understand parts of the conversation without repetition.
- 45 < 60: Unintelligible: Nearly all users are dissatisfied. Frequent instances of long pauses, clicking sounds or distortion can be heard during the call. Frequent repetition is required to be comprehensible, or there are frequent conversation overlaps.
- Under 45 Impossible to communicate
In addition to Voice App Experience, we report on two supporting metrics related to voice app:
- 4G Voice App Experience: This metric quantifies the quality of experience over mobile voice services for each operator on LTE connections as experienced by Opensignal users.
- 3G Voice App Experience: This metric quantifies the quality of experience over mobile voice services for each operator on 3G connections as experienced by Opensignal users.
Download Speed Experience – The real speeds users get
Measured in Mbps, Download Speed Experience represents the typical everyday speeds a user experiences across an operator’s mobile data networks. In addition to Download Speed Experience, we report on three supporting metrics related to download speeds:
- 4G Download Speed: The average downlink speed across an operator’s 4G/LTE connections
- 3G Download Speed: The average downlink speed across an operator’s 3G connections (e.g. UMTS/HSPA or CDMA 1X EV-DO)
- Peak Download Speed: A measure of the fastest download speeds Opensignal users experience on real-world networks. By looking at only the fastest tests we minimize the impact of technical or congestion limitations and instead provide an estimate of the optimal speeds Opensignal users experience in real-world conditions. Note, this is different to the best-case speeds measured in idealized conditions or theoretical maximum speeds that users themselves will never receive in a real-world situation. To assess Peak Speed we use the speed experienced by the 98th percentile of users.
Upload Speed Experience
Our Upload Speed Experience metric measures the average upload speeds for each operator across their mobile data networks. Typically upload speeds are slower than download speeds, as current mobile broadband technologies focus resources on providing the best possible download speed for users consuming content on their devices. As mobile internet trends move away from downloading content to creating content and supporting real-time communications services, upload speeds are becoming more vital and new technologies are emerging that boost upstream capacity.
In addition to Upload Speed Experience, we report on three supporting metrics related to upload speeds:
- 4G Upload Speed: The average uplink speed across an operator’s 4G/LTE connections
- 3G Upload Speed: The average uplink speed across an operator’s 3G connections (e.g. UMTS/HSPA or CDMA 1X EV-DO)
- Peak Upload Speed: A measure of the fastest upload speeds Opensignal users experience on real-world networks. By looking at only the fastest tests we minimize the impact of technical or congestion limitations and instead provide an estimate of the optimal speeds Opensignal users experience in real-world conditions. Note, this is different to the best-case speeds measured in idealized conditions or theoretical maximum speeds that users themselves will never receive in a real-world situation. To assess Peak Speed we use the speed experienced by the 98th percentile of users.
Latency Experience – Quantifying the lag
Latency refers to the delay users experience as data makes a round trip through the network. If the latency of your network is high, the network will feel less responsive and be slower to react to your requests. We measure Latency Experience in milliseconds, and it represents the typical delay a user experiences when connecting across an operator’s 3G and 4G networks. It is calculated as a weighted average of the individual 3G and 4G latencies based on the proportion of time typical users spend connected to each.
All of our download, upload and latency speed tests are run on content delivery networks (CDNs) that host the most popular internet destinations (not on dedicated testing servers where requests might follow a different route to normal traffic or indeed be prioritized in a way that isn’t representative), helping us to accurately understand typical user experience.
In addition to Latency Experience, we report on two supporting metrics related to latency:
- 4G Latency: The average latency users experience across an operator’s 4G/LTE connections
- 3G Latency: The average latency users experience across an operator’s 3G connections (e.g. UMTS/HSPA or CDMA 1X EV-DO
4G Availability – A user-centric approach
Our 4G Availability metric shows the proportion of time users with a 4G device and subscription have an LTE connection. When we report an average 4G availability of 75%, that means our LTE users were, on average, connected to LTE services on their network 75% of the time.
Availability is not a measure of coverage or the geographic extent of a network. It won’t tell you whether you are likely to get coverage if you plan to visit a far flung region that is off the beaten track. Instead, it measures what proportion of time people have a network connection, in the places they most commonly frequent, something often missed by traditional coverage metrics. Looking at when users have an LTE connection, rather than where, provides us with a more precise reflection of the true user experience.
We also keep track of the instances that leave mobile users most frustrated: when there is no signal to connect to at all. The most common dead zones users struggle with occur indoors. As most of our availability data is collected indoors (not surprisingly, since that’s where users spend most of their time), we’re particularly astute at detecting those areas of zero coverage.
4G Coverage Experience
Opensignal’s 4G Coverage Experience metric is a measure of how mobile subscribers experience 4G coverage on an operator’s network. Measured on a scale of 0-10, it analyzes the locations where customers of a network operator received a 4G signal relative to the locations visited by users of all network operators.
In simple terms,4G Coverage Experience measures the mobile coverage experience in all the locations that matter most to everyday users–i.e.all the places where they live, work and travel.-It considers all the areas that Opensignal users visit, the portion of locations that 4G is available to them, and locations that more users visit have higher importance to them.
Feel like checking the numbers for yourself? Download our OpenSignal or Meteor app. Both are, available on Android and iOS!
As of late February 2019, the correct spelling of our company name is Opensignal. OpenSignal is the correct spelling for our app.
Opensignal Limited retains ownership of this insight including all intellectual property rights, data, content, graphs & analysis. Reports and insights produced by Opensignal Limited may not be quoted, reproduced, distributed, published for any commercial purpose (including use in advertisements or other promotional content) without prior written consent. Journalists are encouraged to quote information included in Opensignal reports and insights provided they include clear source attribution. For more information, contact [email protected].