Sign up to our newsletter to receive the latest Opensignal reports in your inbox
Mobile Network Experience Metrics
Definitions Based on Real-World Network Experiences
-
Opensignal’s Video Experience quantifies the quality of video streamed to mobile devices by measuring real-world video streams over an operator's networks.
The metric is based on an International Telecommunication Union (ITU) approach, built upon detailed studies which have derived a relationship between technical parameters, including picture quality, video loading time and stall rate, with the perceived video experience as reported by real people.
To calculate video experience, we are directly measuring video streams from end-user devices and using this ITU approach to quantify the overall video experience for each operator on a scale from 0 to 100.
The videos tested include a mixture of resolutions — including Full HD (FHD) and 4K / Ultra HD (UHD) — and are streamed directly from the world’s largest video content providers.
50
Poor (Under 48)
Our users, on average, encountered very high loading times or high levels of stalling or were only able to stream the video at resolutions below 720p.
Fair (48 or more but less than 58)
Our users were, on average, able to stream video at 720p or better with satisfactory loading times and substantial stalling.
Good (58 or more but less than 68)
Our users were, on average, able to stream video at 720p or better with satisfactory loading times and little stalling.
Very Good (68 or more but less than 78)
Our users were, on average, able to stream video at 1080p or better with satisfactory loading times and little stalling.
Excellent (78 or above)
Our users were, on average, able to stream video at 1080p or better with fast loading times and no stalling.
Use the scale to relate the Video Experience
scores to the actual experience our users received -
Opensignal’s Live Video Experience quantifies the quality of real-time video streamed to mobile devices by measuring video streams over an operator's network. The metric extends the existing International Telecommunication Union (ITU) approach used for Opensignal's on-demand Video Experience metric, built upon detailed studies which have derived a relationship between technical parameters, including live playback offset, picture quality, video loading time and stall rate, with the perceived live video experience as reported by real people.
To calculate live video experience, we are directly measuring live video streams from end-user devices and using this extension of ITU's approach to quantify the overall live video experience for each operator on a scale from 0 to 100.
The videos tested include a mixture of resolutions and are streamed directly from the world’s largest video content providers.
50
Poor (Under 33)
Our users, on average, encountered very high loading times, stalling or live offset or were only able to stream the video at resolutions below 480p.
Fair (33 or more but less than 43)
Our users were, on average, able to stream video at least at 480p with significant loading times, little stalling and a substantial live offset.
Good (43 or more but less than 53)
Our users were, on average, able to stream video at least at 720p with satisfactory loading times, little stalling and a substantial live offset.
Very Good (53 or more but less than 58)
Our users were, on average, able to stream video at least at 720p or 1080p with low loading times, little stalling and a satisfactory live offset.
Excellent (58 or above)
Our users were, on average, able to stream video at least at 1080p with low loading times, little stalling and a satisfactory live offset.
Use the scale to relate the Live Video Experience scores to the actual experience our users received
-
Opensignal’s Games Experience measures how mobile users experience real-time multiplayer mobile gaming on an operator’s network. Measured on a scale of 0-100, it analyzes how our users’ multiplayer mobile gaming experience is affected by mobile network conditions including latency, packet loss and jitter.
Games Experience quantifies the experience when playing real-time multiplayer mobile games on mobile devices connected to servers located around the world. The approach is built on several years of research quantifying the relationship between technical network parameters and the gaming experience as reported by real mobile users. These parameters include latency (round trip time), jitter (variability of latency) and packet loss (the proportion of data packets that never reach their destination).
Additionally, it considers multiple genres of multiplayer mobile games to measure the average sensitivity to network conditions. The games tested include some of the most popular real-time multiplayer mobile games (such as Fortnite, Pro Evolution Soccer and Arena of Valor) played around the world.
Calculating Games Experience starts with measuring the end-to-end experience from users’ devices to internet end-points that host real games. The score is then measured on a scale from 0 to 100.
50
Very Poor (Under 40)
Nearly all users found this level of experience unacceptable. Almost all users experienced a noticeable delay within the game, with most of them not feeling like they had control of the gameplay. The vast majority of players didn’t receive immediate feedback on their actions.
Poor (40 or more but less than 65)
Most users found this level of experience unacceptable. The majority of users reported seeing a delay in the gameplay experience and they did not receive immediate feedback on their actions. Many users felt a lack of controllability.
Fair (65 or more but less than 75)
Users found the experience to be ‘average’. In most cases the game was responsive to the actions of the player with most users reporting that they felt like they had control over the game. The majority of players reported that they noticed a delay between their actions and the outcomes in the game.
Good (75 or more but less than 85)
Most users deemed the experience acceptable. The gameplay experience was generally controllable and the user received immediate feedback between their actions and the outcomes in the game. Most users did not experience a delay between their actions and the game.
Excellent (85 or above)
The vast majority of users deemed this network experience acceptable. Nearly all respondents felt like they had control over the game and they received immediate feedback on their actions. There was not a noticeable delay in almost all cases.
Use the scale to relate the Games Experience
scores to the actual experience our users received -
Opensignal's Voice App Experience measures the quality of experience for over-the-top (OTT) voice services — mobile voice apps such as WhatsApp, FaceTime and Telegram — using a model derived from the International Telecommunication Union (ITU) approach for quantifying overall voice call quality and a series of calibrated technical parameters.
This model characterizes the exact relationship between the technical measurements and perceived call quality. Voice App Experience for each operator is calculated on a scale from 0 to 100.
50
Impossible to communicate (Under 45)
Impossible to communicate.
Unintelligible (45 or more but less than 60)
Nearly all users are dissatisfied. Frequent instances of long pauses, clicking sounds or distortion can be heard during the call. Frequent repetition is required to be comprehensible, or there are frequent conversation overlaps.
Very Poor (60 or more but less than 66)
Most users dissatisfied. Significant call quality impairments experienced by most users. Occasional instances of distortion, clicking sounds or silence experienced during the call. It can be difficult to understand parts of the conversation without repetition.
Poor (66 or more but less than 74)
Many users dissatisfied. Call quality impairments experienced by many users. Distortion, clicking sounds or silence experienced during the call, which is perceptible and can be annoying.
Acceptable (74 or more but less than 80)
Users are satisfied. Perceptible call quality impairments experienced by some users. Short duration of clicking sounds or distortion can be heard, and/or the volume may not be sufficiently loud. Listener is generally able to comprehend without repetition.
Good (80 or more but less than 87)
Many users are satisfied. Minor quality impairments experienced by some users. Sometimes the background is not quite clear, it can be either hazy or not loud enough. Clicking sounds or distortion is very occasionally present.
Very Good (87 or more but less than 95)
Most users are satisfied. Operator generally provides good OTT voice quality experience. Occasionally, there may be some impairments to the call, primarily related to level of loudness.
Excellent (95 or above)
Most users are very satisfied. Operator provides consistently good OTT voice quality experience across the customer base.
Use the scale to relate the Voice App Experience scores to the actual experience our users received -
Opensignal's availability metrics are not measures of a network’s geographical extent. They won’t tell you whether you are likely to get a signal if you plan to visit a remote rural or nearly uninhabited region. Instead, they measure what proportion of time people have a network connection, in the places they most commonly frequent — something often missed by traditional coverage metrics. Looking at when users have a connection rather than where, provides us with a more precise reflection of the true user experience.
We also keep track of the instances that leave mobile users most frustrated: when there is no signal to connect to at all. The most common dead zones users struggle with occur indoors. As most of our availability data is collected indoors (as that’s where users spend most of their time), we’re particularly astute at detecting areas of zero signal.
Opensignal's time on network and availability metrics take a user-centric, time-based approach that complements the user-centric and geographical-based methodology used by our reach metrics.
Our Time on Network metric shows the proportion of time all Opensignal users on an operator’s network had either a 3G, 4G or 5G connection.
Time on 4G shows the proportion of time Opensignal users with a 4G device and a 4G subscription — but have never connected to 5G — had a 4G connection.
Time on 5G shows the proportion of time Opensignal users with a 5G device and a 5G subscription utilized an active 5G connection.
5G Availability shows the proportion of time Opensignal users with a 5G device and a 5G subscription had a 5G connection, whether or not it was used.
-
The Opensignal Coverage Experience metric measures the extent of mobile networks in the places people live, work and travel. The metric represents the experience users receive as they travel around areas where they would reasonably expect to find coverage.
Traditional coverage metrics typically estimate either a percentage of land area covered, or a percentage of population covered; often neither will be an accurate measurement of the true user expectation and experience. In many markets there are areas where neither population density nor geographic area reflect the importance of coverage to users.
For example, in a large mountain range most users will not expect coverage in the wilderness, but poor coverage in the relatively small area of a ski resort is critical for the enjoyment of a holiday. Estimates based purely on population give undue significance to coverage in the most densely populated areas.
Coverage Experience measures geographic coverage of populated areas and therefore more accurately reflects the coverage expectations and experience of typical users. It can give a result that is somewhat different to traditional estimates based on either geographic or population measures. The metric uses a scale from 0 to 10.
-
Consistent Quality measures if the network is sufficient to support common mobile application requirements at a level that is ‘good enough’ for users to maintain (or complete) various typical tasks on their devices.
We combine different experience indicators such as download throughput, upload throughput, latency, jitter, packet discard, and time to first byte to calculate Consistent Quality. These components are evaluated against thresholds recommended by various more demanding common applications used for a range of common tasks.
To calculate the metric value, the proportion of tests that pass the requirements of Consistent Quality is multiplied by the test success ratio, which is the proportion of completed tests to all tests conducted. Tests that pass indicate that activities such as video calling, uploading an image to social media, or using smart home applications will be possible without noticeable lag or slowdown.
-
Opensignal’s Reliability Experience measures the ability of Opensignal users to connect to and successfully complete (basic) tasks on communication service providers’ (CSP) networks. It analyzes how much Opensignal users’ experience is affected by the radio access and core network, along with issues that prevent them from connecting to the internet even if they have a connection to their CSP’s network. It also factors in users’ ability to successfully use lower performance applications including SD video, over-the-top voice calls and web browsing.
It does not test standard voice calls performed directly through CSPs’ networks — i.e. those that are carried out via smartphones’ in-built call functionality.
Calculated on a scale of 100-1000 — with higher scores being better — across all generations of mobile technology, Reliability consists of the following components:
- % time connected — The proportion of time Opensignal users can successfully connect to a mobile network (if appropriate)
- (Data) Connectivity — the proportion of time when the network is available and the device can connect to the internet
- Task completion — whether tasks initiated by the user’s device are completed. Completion is measured on a scale of 0% to 100%. It is measured with an active UDP test to a test server.
- Sufficiency — The probability that (basic) tasks will be executed sufficiently well for the user. The percentage of tests that pass all the required thresholds of Sufficiency indicating whether users have a sufficient session when they are able to connect to the network on completed tasks. Sufficiency is calculated on a 0% to 100% scale. If any threshold is not met for a given testing session, the Sufficiency for that session is 0%. The required thresholds for a successful Sufficiency test are:
- Download throughput > 1.5 Mbps
- Upload throughput > 0.5 Mbps
- Latency < 100ms
- Jitter < 20ms
- Packet discard < 4%
- Time to first byte < 1.2s
-
Measured in Mbps, Opensignal's Download Speed Experience represents the typical everyday speeds a user experiences across an operator’s mobile data networks.
- 5G Download Speed
The average download speed observed by Opensignal users with active 5G connections.
- Download Speed Experience – 5G Users
The average download speeds experienced by Opensignal users with a 5G device and a 5G subscription across an operator’s networks. It factors in 2G, 3G, 4G, and 5G download speeds along with the availability of each technology.
- 4G Download Speed
The average downlink speed observed by Opensignal users when they were connected to 4G.
- 3G Download Speed
The average downlink speed observed by Opensignal users when they were connected to 3G (e.g. UMTS/HSPA or CDMA 1X EV-DO).
- 5G Download Speed
-
Measured in Mbps, Opensignal's Upload Speed Experience measures the average upload speeds for each operator observed by our users across their mobile data networks.
- 5G Upload Speed
The average upload speed observed by Opensignal users with active 5G connections.
- Upload Speed Experience – 5G Users
The average upload speeds experienced by Opensignal users with a 5G device and a 5G subscription across an operator’s networks. It factors in 2G, 3G, 4G, and 5G upload speeds along with the availability of each technology.
- 4G Upload Speed
The average uplink speed observed by Opensignal users when they were connected to 4G.
- 3G Upload Speed
The average uplink speed observed by Opensignal users when they were connected to 3G (e.g. UMTS/HSPA or CDMA 1X EV-DO).
- 5G Upload Speed
-
The Best Network designation can be recognized for operators that provide an excellent user experience and have a strong presence. To qualify, an operator must:
1. Be an outright winner
In at least two of the following three core performance categories:
- Download Speed Experience
- Consistent Quality
- Reliability Experience
If no operator meets this exact criterion, we consider those with two joint wins and one outright win across the same three categories.
AND
2. Also be a joint or outright winner
In one of the two coverage-related categories:
- Coverage Experience, or
- Time on Network
This methodology highlights operators that perform well across key experience metrics while also ensuring widespread access to their mobile services.
It can be applied to all technology types and geographies where sufficient samples are available to determine winners for the individual categories.
Sign up to our newsletter to receive the latest Opensignal reports in your inbox