While this post is aimed at Web performance, the curse of the single metric affects our everyday lives in ways that we have become oblivious to.

When you listen to a business report, the stock market indices are an aggregated metric used to represent the performance of a set group of stocks.

When you read about economic indicators, these values are the aggregated representations of complex populations of data, collected from around the country, or the world.

Sport scores are the final tally of an event, but they may not always represent how well each team performed during the match.

The problem with single metrics lies in their simplicity. When a single metric is created, it usually attempts to factor in all of the possible and relevant data to produce an aggregated value that can represent a whole population of results.

These single metrics are then portrayed as a complete representation of this complex calculation. The presentation of this single metric is usually done in such a way that their compelling simplicity is accepted as the truth, rather than as a representation of a truth.

In the area of Web performance, organizations have fallen prey to this need for the compelling single metric. The need to represent a very complex process in terms that can be quickly absorbed and understand by as large a group of people as possible.

The single metrics most commonly found in the Web performance management field are performance (end-to-end response time of the tested business process) and availability (success rate of the tested business process). These numbers are then merged and transformed by data from a number of sources (external measurements, hit counts, conversions, internal server metrics, packet loss), and this information is bubbled up in an organization. By the time senior management and decision-makers receive the Web performance results, that are likely several steps removed from the raw measurement data.

An executive will tell you that information is a blessing, but only when it speeds, rather than hinders, the decision-making process. A Web performance consultant (such as myself) will tell that basing your decisions on a single metric that has been created out of a complex population of data is madness.

So, where does the middle-ground lie between the data wonks and the senior leaders? The rest of this post is dedicated to introducing a few of the metrics that will, in a small subset of metrics, give a senior leaders better information to work from when deciding what to do next.

A great place to start this process is to examine the percentile distribution of measurement results. Percentiles are known to anyone who has children. After a visit to the pediatrician, someone will likely state that “My son/daughter is in the XXth percentile of his/her age group for height/weight/tantrums/etc”. This means that XX% of the population of children that age, as recorded by pediatricians, report values at or below the same value for this same metric.

Percentiles are great for a population of results like Web performance measurement data. Using only a small set of values, anyone can quickly see how many visitors to a site could be experiencing poor performance.

If at the median (50th percentile), the measured business process is 3.0 seconds, this means that 50% of all of the measurements looked at are being completed in 3.0 seconds or less.

If the executive then looks up to the 90th percentile and sees that it’s at 16.0 seconds, it can be quickly determined that something very bad has happened to affect the response times collected for the 40% of the population between these two points. Immediately, everyone knows that for some reason, an unacceptable number of visitors are likely experiencing degraded and unpredictable performance when they visit the site.

A suggestion for enhancing averages with percentiles is to use the 90th percentile value as a trim ceiling for the average. Then side-by-side comparisons of the untrimmed and trimmed averages can be compared. For sites with a larger number of response time outliers, the average will decrease dramatically when it is trimmed, while sites with more consistent measurement results will find their average response time is similar with and without the trimmed data.

It is also critical to examine the application’s response times and success rates throughout defined business cycles. A single response time or success rate value eliminates

• variations by time of day
• variations by day of week
• variations by month
• variations caused by advertising and marketing

An average is just an average. If at peak buiness hours, response times are 5.0 seconds slower than the average, then the average is meaningless, as business is being lost to poor performance which has been lost in the focus on the single metric.

All of these items have also fallen prey to their own curse of the single metric. All of the items discussed above aggregate the response time of the business process into a single metric. The process of purchasing items online is broken down into discrete steps, and different parts of this process likely take longer than others. And one step beyond the discrete steps are the objects and data that appear to the customer during these steps.

It is critical to isolate the performance for each step of the process to find the bottlenecks to performance. Then the components in those steps that cause the greatest response time or success rate degradations must be identified and targeted for performance improvement initiatives. If there are one or two poorly performing steps in a business process, focusing performance improvement efforts on these is critical, otherwise precious resources are being wasted in trying to fix parts of the application that are working well.

In summary, a single metric provides a sense of false confidence, the sense that the application can be counted on to deliver response times and success rates that are nearly the same as those simple, single metrics.

The average provides a middle ground, a line that says that is the approximate mid-point of the measurement population. There are measurements above and below this average, and you have to plan around the peaks and valleys, not the open plains. It is critical never to fall victim to the attractive charms that come with the curse of the single metric.

I try and avoid the “me-too” factor that has dominated the land of blogs for most of the time I have been involved in it. Simply aping one persons comments with a slight variation, or personal interpretation doesn’t add much to the initial thrill of finding the original germ of an idea.

Kathy Sierra, someone who has been quoted and analyzed multiple times in this blog, has hit another double to the wall. She talks about the value of serendipity, randomness, in exposing us to new ideas and concepts, ones that we would not have run across in our siloed, standardized lives.

Yesterday was a great example of this for me. Something I read a post on Notebookism that spoke of outsider art or Art Brut. I looked it up on Wikipedia, and spiralled into a 90-minute voyage of discovery into this genre of expression, fueled not by training and ideology, but by a raw, unchecked need to express the world in an artistic way.

I would have never gone down this path unless I had read the Notebookism post, and would have been hard-pressed to find structured explanations (whatever you may think of them) of the topics without Wikipedia.

As I explore myself, and examine the foundations that support my cracked mental structure, I find that I appreciate the random explorations far more than a formal education process. I don’t learn the way that we have been taught.

I prefer to discover.

And when you get right down to the basics of Kathy’s post, that’s what she is saying. People are far more enthusiastic, receptive, and amazed when they discover something for themselves.

It may be an old idea to you. I may not interest you. But when a person gets that gleam in their eye, that rush in their mind, when they get the “WOW!“, then they are committed.

Personally, I am finding that I am having a lot more WOW! moments lately. The combination of therapy, and my medications, has forced me to look at the world that I live in, and the world that I have created, substantially different than I have for the last 15 years.

I am re-discovering the joy and awe of discovery. There is so much out there that gets left behind when your mind is absorbed, consumed, by a single devouring purpose. I am awakening from that period, and finding that my mental indigestion requires the soothing relief of the new and unexpected.

I have talked in other posts about being an accumulator, driven by the mania to buy things that I have no need for, nor will I ever use. A slight variation of this theme is the need to hoard.

How can accumulation be different from hoarding? It’s simple, it’s not simply the accumulation of things; it’s the hoarding of them in caches, stores, and never share them with anyone.

As well as accumulation, I hoard. I am loathe to throw anything away. I must have it there, in case I need it, sometime, anytime.

Files. Papers. Photographs. Gadgets. Their mine. All mine. Don’t touch them! Get out of my space!

It is another inexplicable part of the Bipolar. The manic need to keep it all close. To protect it from others. To keep them from taking your irrationally collected things.

Isn’t the mind a fun place?

Yahoo Search Blog has a great posting on how the Slurp Bot tries to conserve bandwidth by making use of compression and cache-control headers. [here]

As a Web performance fanatic, it is heartening to see that these folks have taken such care, and put such thought into their indexing crawler. They want it to be accurate, but they don’t want to slam your site.

A while back, I had to write a robots.txt file for WebPerformance to keep the MSNBot from stomping the site on a daily basis. This site uses frames and query variables to produce the various performance graphs. Well, the MSNBot was indexing every page and every variation almost daily. Finally, I said go away, just to that crawler. All the others are fine. Maybe MSN Search should take a page from the Yahoo! (Inktomi) Bot development team.