Insights Built on Real-World Network Experiences

When people use mobile and broadband connections, they care most about how they are able to communicate, play games or consume content. We measure the real-world experience of consumers as they use these networks, collected independent of any carrier-specific data feeds.

Our tests are designed to measure the real-world experience of users

We compare network experiences based on measures that matter to customers as they use their devices across connected activities. We focus on this anonymous, user-centric approach to reflect how all consumers engage, within their network and compared to other networks that operate in the same locations.

Real-world insights ...

Our methodologies related to network experience comprise the real world activity of anonymous research participants as they use mobile and broadband networks, which are captured and used to develop our proprietary insights.

... that enable impactful decisions

These insights inform our public market reports and help network operators, regulators, analysts and consumers make decisions on how to connect, invest, and assess networks in their markets.

View Market Insights

Review our experience metrics

Mobile Network Experience Metrics

Review Here

Broadband Network Experience Metrics

Review Here

Proudly independent

The insights we present are collected independently using anonymous data collection processes. The data science we use to power our insights and comparisons are only published when they are statistically valid.

Read our charters

How we approach our metrics and insights

How Opensignal collects and analyzes data

Data collection methods

Our process begins with collecting billions of measurements daily from a diverse set of research participants covering a wide range of users, demographics, and devices.

Superior data accuracy

Our insights come from a diverse and representative panel. With consent, our data collection process is automated, tests run all the time, in all the places people live, work and play. Indoor and outdoors, irrespective of whether participants connect over wi-fi or cellular connections, providing the largest and most representative data set.

Use of Common Internet Endpoints

Opensignal tests are conducted to common internet endpoints including Content Delivery Networks (CDNs) such as Google, Akamai and Amazon. This is identical to the way connections are made each day to typical websites and content and provides a truly representative end-to-end view of network experience.

Key Differentiators

Our measurements are designed so that operators cannot optimize their networks to treat our traffic differently and therefore impact the results without making actual improvements to their networks. This approach is unique to Opensignal, and supports the needs of consumers who spend their time using applications like video and games, providing network operators with actionable insights that support improvements based on how consumers actually use their digital devices.

Privacy of our users

We take extensive measures to ensure that the privacy of research participants is respected through the entire process and that data is only used for the depersonalized purposes that they permit when they opt-in. The details can be found in our Data Privacy Charter.

Data analysis

The processing and analysis for all of the measurements we collect is based on best-in-class data science methods following the principles of our Experience Charter and our Analytics Charter and is explained in more detail below.

The process is designed to ensure that incorrect or potentially distorting data is not able to influence the results and that results are shown in a way that can be clearly understood and relied upon.

Our approach to testing removes the potential for bias

Opensignal collects measurements of network experience quality and speed based on both user-initiated and regular periodic tests.

The majority of measurements are generated through regularly scheduled periodic tests, executed independently and at random intervals to capture what users are experiencing at a typical moment in time. This approach is recognized as best practice by a number of official bodies including the FCC in the U.S.

Active speed measurement

Finding the stable speed is what matters

When any application downloads data there is an initial ramp-up period as the connection is established where the download speed will not be the same as the stable speed achieved once the download is in progress.

A consumer's experience on today's networks is based on speed-sensitive applications such as streaming video or large file downloads. These are influenced mainly by the stable speed (sometimes called the “goodput”) throughout the task, not the initial speed when the action is triggered. Conversely applications such as web browsing are influenced heavily by ramp-up time and latency. Our speed tests look at the need to accomplish a task rather than the test of speed as a standalone metric.

Our independent, automated approach to network experience testing is considered an official best practice.

Using fixed time versus fixed file size allows comparisons to be made across varying speeds, technologies and tasks.

Fixed time tests mimic user perception

We use a fixed time test since our focus is on measuring the user experience of applications such as streaming and large file downloads. As these experiences are influenced primarily by the stable speed, we use a fixed time test, rather than a simple fixed file size download. A fixed time period enables the speed measured to be a much closer representative of the stable speed the user experiences through the application.

As well as being more representative, this fixed time approach makes comparisons between widely different network speeds more meaningful. A file of a few MB downloaded over a 3G network will take several seconds and the speed measured will be influenced primarily by the goodput and only slightly by the ramp-up. The download of the same file over an 5G network will take a much shorter time and the overall speed measured will be dominated by the ramp-up time and unrelated to the speed seen by a streaming application.

Video Experience measurement

Direct streams from CDN to device

The Opensignal Video Experience measurement directly streams sample video from typical content providers and measures a range of parameters that directly impact the user experience, such as the loading time (the time taken for the video to start) and the video stalling ratio (the proportion of users who experience an interruption in playback after video begins streaming) for different picture qualities or bit rates.

Real experience from real streams

To calculate our video metric, we use video content providers selected to represent typical user experience.

Reliability, Quality and Foundational Experiences

Connecting to the Network and Completing Tasks

Our Reliability and Consistent Quality (CQ) measurements capture the foundational experiences without which consumers remain frustrated. Importantly we are measuring both a user's ability to connect to the internet (not just the signal availability) and their ability to perform common, foundational tasks when they are connected.

We use the TCP and UDP protocols to measure round trip traffic to internet end points and capture internet fundamentals like connectivity, latency, packet loss and jitter.

Our measurements are designed to prevent the potential for a result to be biased.

Processing data for statistical significance

Opensignal uses a rigorous post-processing system that takes the raw measurements and calculates robust and representative metrics. This includes a number of steps to quality-assure the measurements.

For example, if a user failed to download any content, this measurement is eliminated and treated as a “failed test” rather than being included in the average speed calculation.

Similarly, when calculating metrics on a given network technology (e.g. 4G), measurements where a network type change is detected (e.g. from 5G to 4G) during the duration of the measurement are not included.

Some additional rules we follow include:

Initial filtering

We automatically filter out certain entries, (e.g. when a phone is in a call) which are known to produce non-typical results.

Operator name mapping

To ensure that the results only reflect the experience of customers who bought the operator’s own branded service, we treat separately results from Mobile Virtual Network Operator (MVNO) subscribers and subscribers who are roaming to those of the Mobile Network Operators. These subscribers may be subject to different Quality of Service (QoS) restrictions than an operator’s own customers and so their experience may be different.

For fixed broadband connections we consider the consumer facing brands in our reporting, unless we stated otherwise. Where wholesale infrastructure exists we consider the results for individual brands as peering connections and CPE selections can have significant impact on the customer experience.

Selection of network type

We consolidate data into technology types — e.g. when considering 5G connections, we include mmW, low band and mid-band connections into a single technology type unless stated otherwise.

Scientific averaging

We calculate a single average per samping device to ensure every device has an equal effect on the overall result. Essentially, we employ a “one device, one vote” policy in our calculations.

Removing extreme values

We eliminate a percentage of extreme high and low values. This removal of extremes is common data science practice and ensures the average calculated represents typical user experience.

How we report our analysis

Charters