Methodology Overview

Opensignal’s objective is to report as accurately as possible the real-world mobile experience as recorded by mobile network users.

We strongly believe that:

• What matters most when assessing network performance is how it is experienced by subscribers themselves. This is best understood by capturing the real-world experience of users as they interact with the network in the places where they live, work and travel.
• By making this information transparently available, we address the important need for consumers to understand the most relevant aspect of network performance relative to them, namely how it translates to their user experience

Although operators have been monitoring performance from when the very first networks were built, there remains a disconnect between their engineering tests and the typical experience of everyday users.

Opensignal’s public reports contain the set of metrics we believe best represent users’ experience of accessing real services. We continually invest in research and development to find new measures. For example, in 2018 we launched the telecom industry’s first independent measure of real-world mobile video experience, spanning multiple video content platforms.

Our process begins with collecting billions of measurements daily from over 100 million devices globally.

Our software is installed within our own and partner apps. The partners we work with are strategically selected to cover a wide range of users, demographics, and devices.

Some speed tests published by other sources use data only from their own apps, which restricts the sample to a certain type of user, and some restrict data collection to just the latest devices, which again limits the user base.

• Opensignal collects billions of individual measurements every day from over 100 million devices worldwide, under conditions of normal usage, including measurements in both indoor and outdoor locations. Users spending most of their times in indoor locations mean that most of our measurements will be collected from indoor locations.
• To calculate our video metric, we use video content providers selected to represent typical user experience. Our measurements are designed so that operators cannot optimize their networks to treat our traffic differently and therefore impact the results without making actual improvements to their networks.
• Opensignal collects measurements of network speed based on both user-initiated tests and automated tests. The majority of measurements are generated through automated tests (no user interaction), executed independently and at random intervals to capture what users are experiencing at a typical moment in time. This approach is recognized as best practice by a number of official bodies including the FCC in the U.S.

When any application downloads data there is an initial ramp-up period as the connection is established where the download speed will not be the same as the stable speed achieved once the download is in progress.

On today’s networks, from a user experience point of view, speed-sensitive applications such as streaming video or large file download are influenced mainly by the stable speed (sometimes called the “goodput”), not the speed during the ramp-up time. Conversely applications such as web browsing are influenced heavily by the ramp-up time and by latency.

Measuring Speed Experience: As the Opensignal Download Speed Experience metric is focused on measuring the user experience of these applications such as streaming and large file downloads, which are influenced primarily by the stable speed, we use a fixed time test, rather than a simple fixed file size download. A fixed time period enables the speed measured to be a much closer representative of the stable speed the user experiences through the application.

The Opensignal Video Experience measurement directly streams sample video from typical content providers and measures a range of parameters that directly impact the user experience, such as the loading

Opensignal uses a rigorous post-processing system that takes the raw measurements and calculates robust and representative metrics. This includes a number of steps to quality-assure the measurements.

For example, if a user failed to download any content, this measurement is eliminated and treated as a “failed test” rather than being included in the average speed calculation.

Per Device Values: Per device values are combined using a simple average to yield the Opensignal metrics that are found in our reports and analysis. 

Confidence Intervals: We provide an upper and lower estimate of confidence interval per operator, calculated using recognized standard techniques based on the sample size of measurements. Confidence intervals provide information on the margins of error or the precision in the metric calculations. They represent the range in which the true value is very likely to be, taking into account the entire range of data measurements.

Statistically significant results: Whenever the confidence intervals for two or more operators overlap in a particular metric, the result is a statistical tie. This is because one operator's apparent lead may not hold true once measurement uncertainty is taken into account. For this reason, awards could have multiple winners in our reports.

Opensignal’s Video Experience quantifies the quality of video streamed to mobile devices by measuring real-world video streams over an operator's networks. The metric is based on an International Telecommunication Union (ITU) approach, built upon detailed studies which have derived a relationship between technical parameters, including picture quality, video loading time and stall rate, with the perceived video experience as reported by real people.

To calculate video experience, we are directly measuring video streams from end-user devices and using this ITU approach to quantify

Opensignal’s Games Experience measures how mobile users experience real-time multiplayer mobile gaming on an operator’s network. Measured on a scale of 0-100, it analyzes how our users’ multiplayer mobile gaming experience is affected by mobile network conditions including latency, packet loss and jitter.

Games Experience quantifies the experience when playing real-time multiplayer mobile games on mobile devices connected to servers located around the world. The approach is built on several years of research quantifying the relationship between technical network parameters and the gaming experience as reported by real mobile users. These parameters include latency (round trip time), jitter (variability of latency) and packet loss (the proportion of data packets that never reach their destination).

Opensignal's Voice App Experience measures the quality of experience for over-the-top (OTT) voice services — mobile voice apps such as WhatsApp, Skype and Facebook Messenger — using a model derived from the International Telecommunication Union (ITU) approach for quantifying overall voice call quality and a series of calibrated technical parameters.

Group Video Calling Experience measures the proportion of video calls where all users on a call had at least an adequate or better video conference experience.

In simple terms, Group Video Calling Experience measures whether all users in a group video call – not just a small number of users – had both sufficient (or better) video and audio quality. It therefore takes into account that a poor experience for one or more users will impact all users on a conference call so having a consistent experience across all users on a group video call is important.

Opensignal's availability metrics are not measures of a network’s geographical extent. They won’t tell you whether you are likely to get a signal if you plan to visit a remote rural or nearly uninhabited region. Instead, they measure what proportion of time people have a network connection, in the places they most commonly frequent — something often missed by traditional coverage metrics. Looking at when users have a connection rather than where, provides us with a more precise reflection of the true user experience. 

We also keep track of the instances that leave mobile users most frustrated: when there is no signal to connect to at all. The most common dead zones users struggle with occur indoors. As most of our availability data is collected indoors (as that’s where users spend most of their time), we’re particularly astute at detecting areas of zero signal.

Opensignal’s reach metrics measure how mobile users experience the geographical extent of an operator’s network. They analyze the average proportion of locations where users were connected to a network out of all the locations those users have visited. 

In simple terms, reach metrics measure the mobile experience in all the locations that matter most to everyday users — i.e. all the places where they live, work and travel. Our reach metrics for each operator are measured on a scale from 0 to 10.

Opensignal’s reach metrics provide a user-centric, geographical-based measure to complement the user-centric, time-based measure provided by availability metrics.

4G Coverage Experience measures how mobile subscribers experience 4G coverage on an operator’s network. Measured on a scale of 0-10, it analyzes the locations where customers of a network operator received a 4G signal relative to the locations visited by users of all network operators.

In simple terms, 4G Coverage Experience measures the mobile coverage experience in all the locations that matter most to everyday users — i.e. all the places where they live, work and travel. It considers all the areas

Consistent Quality measures how often users' experience on a network was sufficient to support common applications' requirements. It measures download speed, upload speed, latency, jitter, packet loss, time to first byte and the percentage of tests attempted which did not succeed due to a connectivity issue on either the download or server response component. 

Full details on how the Consistent Quality metrics – Excellent Consistent Quality and Core Consistent Quality - are measured can be found here.

Our measurements are designed to capture as accurately as possible the experience of typical real users and are subject to all the factors that affect real user traffic, representing a wide base of users and devices.

Our scientific analysis processes these measurements to create the most accurate possible picture of user experience and how it varies between operators, regions and countries, as far as possible based only on measurements from real users.