Stop using averages. Use percentiles instead

Bill Gates walks into a pub. Instantly, the average wealth of each person in the pub is 45 million dollars.

I’ve never been in a pub with Bill Gates, but I imagine everyone else would join in in disagreement with that statement.

But really, averages are bad

There are plenty of books and articles written on this very topic, and I don’t claim to know more than any of the information I sourced to write this. But let’s start by looking at what averages represent, and how data works in real life.

The primary purpose of averages is to measure changes over time in the same sample group or cohort. It is in this application, or more so misapplications, by using averages for different purposes that the three most common errors occur.

Most sets of data have outliers. More often than not, extreme outliers. Let’s see an initial example:

Using Percentiles

Percentiles are a fantastic way to avoid data skewing, because you’ll have more metrics to look at which happen to be representative of the experiences of that data.

Our average, 559ms, misses the mark entirely

Calculating percentiles

You probably already know how to calculate a p50 percentile, because that’s really just the median: you sort the data is ascending order (like our set above) and you discard the bottom 50% of your points. Your next entry point is your p50 value. So above, we throw away half of the values and we’re left with:

  • p50 values = [ ̶1̶2̶0̶,̶ ̶1̶2̶0̶,̶ ̶1̶3̶4̶,̶ ̶1̶5̶5̶, 300, 867, 980, 1800]
  • p90 values = [ ̶1̶2̶0̶,̶ ̶1̶2̶0̶,̶ ̶1̶3̶4̶,̶ ̶1̶5̶5̶,̶ ̶3̶0̶0̶,̶ ̶8̶6̶7̶,̶ ̶9̶8̶0̶, 1800]

What percentiles tell us

Imagine you’re feeling pretty good about your 559ms average value. One morning, a few reports from customer support come in warning your team about some user frustrations: some users are experiencing page load times of almost 2 seconds. It’s easy to dismiss it, because after all your average is well under 600ms. But what percentile calculation gives you is the knowledge that 90% of your users are having an experience that is lower than 2 seconds; but the bottom 10% aren’t really happy at all.

p50 and p90 tell a more accurate story

Settings goals

Using percentiles allows gives us much more valuable information. Instead of focusing on averages, an example performance goal for your team can now be:

Downsides to all this

While there are frameworks and tools to help you calculate these values over time, these calculations tend to be computationally heavy and demand extra resources, both in terms of servers and costs. If you’re saving every piece of information (page load time, request time, etc) then you’ll have a ton of data on your hands. Most tools will allow you to tweak memory vs accuracy ratios, because fundamentally percentiles are expensive approximations.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ricardo Magalhães

Ricardo Magalhães

I work for the Internet. By day, a front-end web developer with a passion for typography and design. By night, I’m sleeping.