While this post is aimed at Web performance, the curse of the single metric affects our everyday lives in ways that we have become oblivious to.

When you listen to a business report, the stock market indices are an aggregated metric used to represent the performance of a set group of stocks.

When you read about economic indicators, these values are the aggregated representations of complex populations of data, collected from around the country, or the world.

Sport scores are the final tally of an event, but they may not always represent how well each team performed during the match.

The problem with single metrics lies in their simplicity. When a single metric is created, it usually attempts to factor in all of the possible and relevant data to produce an aggregated value that can represent a whole population of results.

These single metrics are then portrayed as a complete representation of this complex calculation. The presentation of this single metric is usually done in such a way that their compelling simplicity is accepted as the truth, rather than as a representation of a truth.

In the area of Web performance, organizations have fallen prey to this need for the compelling single metric. The need to represent a very complex process in terms that can be quickly absorbed and understand by as large a group of people as possible.

The single metrics most commonly found in the Web performance management field are performance (end-to-end response time of the tested business process) and availability (success rate of the tested business process). These numbers are then merged and transformed by data from a number of sources (external measurements, hit counts, conversions, internal server metrics, packet loss), and this information is bubbled up in an organization. By the time senior management and decision-makers receive the Web performance results, that are likely several steps removed from the raw measurement data.

An executive will tell you that information is a blessing, but only when it speeds, rather than hinders, the decision-making process. A Web performance consultant (such as myself) will tell that basing your decisions on a single metric that has been created out of a complex population of data is madness.

So, where does the middle-ground lie between the data wonks and the senior leaders? The rest of this post is dedicated to introducing a few of the metrics that will, in a small subset of metrics, give a senior leaders better information to work from when deciding what to do next.

A great place to start this process is to examine the percentile distribution of measurement results. Percentiles are known to anyone who has children. After a visit to the pediatrician, someone will likely state that “My son/daughter is in the XXth percentile of his/her age group for height/weight/tantrums/etc”. This means that XX% of the population of children that age, as recorded by pediatricians, report values at or below the same value for this same metric.

Percentiles are great for a population of results like Web performance measurement data. Using only a small set of values, anyone can quickly see how many visitors to a site could be experiencing poor performance.

If at the median (50th percentile), the measured business process is 3.0 seconds, this means that 50% of all of the measurements looked at are being completed in 3.0 seconds or less.

If the executive then looks up to the 90th percentile and sees that it’s at 16.0 seconds, it can be quickly determined that something very bad has happened to affect the response times collected for the 40% of the population between these two points. Immediately, everyone knows that for some reason, an unacceptable number of visitors are likely experiencing degraded and unpredictable performance when they visit the site.

A suggestion for enhancing averages with percentiles is to use the 90th percentile value as a trim ceiling for the average. Then side-by-side comparisons of the untrimmed and trimmed averages can be compared. For sites with a larger number of response time outliers, the average will decrease dramatically when it is trimmed, while sites with more consistent measurement results will find their average response time is similar with and without the trimmed data.

It is also critical to examine the application’s response times and success rates throughout defined business cycles. A single response time or success rate value eliminates

• variations by time of day
• variations by day of week
• variations by month
• variations caused by advertising and marketing

An average is just an average. If at peak buiness hours, response times are 5.0 seconds slower than the average, then the average is meaningless, as business is being lost to poor performance which has been lost in the focus on the single metric.

All of these items have also fallen prey to their own curse of the single metric. All of the items discussed above aggregate the response time of the business process into a single metric. The process of purchasing items online is broken down into discrete steps, and different parts of this process likely take longer than others. And one step beyond the discrete steps are the objects and data that appear to the customer during these steps.

It is critical to isolate the performance for each step of the process to find the bottlenecks to performance. Then the components in those steps that cause the greatest response time or success rate degradations must be identified and targeted for performance improvement initiatives. If there are one or two poorly performing steps in a business process, focusing performance improvement efforts on these is critical, otherwise precious resources are being wasted in trying to fix parts of the application that are working well.

In summary, a single metric provides a sense of false confidence, the sense that the application can be counted on to deliver response times and success rates that are nearly the same as those simple, single metrics.

The average provides a middle ground, a line that says that is the approximate mid-point of the measurement population. There are measurements above and below this average, and you have to plan around the peaks and valleys, not the open plains. It is critical never to fall victim to the attractive charms that come with the curse of the single metric.

Tags: business process, components, curse, cycles, metrics, percentiles, performance, statistics, steps, trim, truth, variation, web, Web Performance, Web performance management, Web performance management field, Web performance measurement data, Web performance results

The last 4 weeks have been extremely traumatic for me. It has culminated in an extended period of renewal, reflection and rejuvenation, where I have looked back over the last 15 years of my life and asked, “What next?”.

An interesting note on the word rejuvenation: it means to reclaim your childlike state (ok, I’m playing fast and loose with the definition).

Why now? Why 15 years?

In the Fall of 1991, I bought my first computer. Until then, I had avoided using them like the plague. I had managed to get through my undergraduate years with a pen, paper, and an electronic typewriter with rudimentary spell-checking. I felt that I had achieved something; I felt bonded to the works I created.

I was also an avid and active journal-keeper. In the months after my father died, the writing in my journal was what let me empty my naive mind, letting me vent the chaos that rushed through my head on a constant basis.

Then I went to grad school. And I realized then that I would need to step up in order to generate the massive amount of paper that is required in a graduate history program.

It turns out that I found the technology more enticing than the program. To this day, my failure to complete my Masters degree haunts me. Someday, I will return to that, and complete it. Knot the loose ends of my life together.

Ok, this really is going somewhere; thanks for hanging on this far.

After 15 years of intense immersion in technology, the Web, networking, and all that comes along with that, I have realized that something has been missing from my work, my writing, my life. I have missed the rushing sound of pen on a clean sheet of blank paper. No lines to slow you down; nothing besides the edges of the page to define what you put in the book.

Technology has lost its lustre. The rushing stream of this new laptop, that new technology, another over-inflated boom have left me feeling empty, asking “So what?”. In a hundred years, we can be so far down the path to post-humanism that computers as we know them are a vague and distant antique amusement.

Or we could be living in caves, scratching by a subsistence existence.

In either case, the only thing that will remain, that will linger, that will connect us to the past will be the written word. Not the electronic bits and bytes we are now so addicted to, but the ink on paper, graphite on wood pulp.

The smooth, quiet, seductive transition of ideas from mind to physical reality.

I have been trimming back my blog-reading. Gone are the political blogs. I fear that the gadget blogs are next.

What you have left are those people who celebrate life outside the electronic realm. Those who step back, and look back on the knowledge that preceded us. Who pick up a book that was published before they were born.

A book that left the mind of the author and flowed gracefully from the pen, to the paper, to another mind.

15 years is a long time to try and live without paper. Those 15 years have seen the niceties of a bygone age evaporate, get swallowed by an endless sea, a raging torrent of information.

The cursive hand; the thoughtful response; the flowing of ideas from person to person.

To calm the storm of my mind, I have returned to my first love: ideas of the mind, of the soul. Ideas that were worthy of the preparation of the parchment, the sharpening of the quill, the grinding of the pigment to create the ink.

We have walked away from those ideas, grasping at the brass ring in front of us, to the disdain of the treasure chest we leave behind.

To focus on the ideas, that is to live again.

To heal my mind, I must write my mind. Not type it; not IM it or e-mail it or blog it.

That familiar scratch of pen on paper. The rush that comes from committing something to paper; something that you can share with others.

Something that you can set adrift, watch as it floats, the glow from its candle on the gentle rippled flow of all the ideas that have come before.

I am setting my ideas free again.

Picture: girlzone41

Technorati Tags: ideas, quill, ink, pigment, death, life, resurrection, rebirth, technology, paper, pen, rejuvenation, myths, legends, chaos, healing, mind

Tags: ASP, blog, book, EAD, HTTP, IE, IM, information, IP, it, LAN, leave, live, networking, Om, One, pr, Technorati, Torrent, trim, web

Netcraft reported that World of Warcraft was having a wee bit o’ performance hell. [here]

Guess a few wizards are going through withdrawal right now.

Looks like Joi Ito, a serious World of Warcraft fiend, might have to mount up his troops for an all out assault on this one!

PS: Looks like it just staggered to life! Watch the onhoing saga here!

Technorati Tags: Netcraft, World of Warcraft, Joi Ito, GrabPERF, Web Performance

Tags: ARIN, EAD, Google, GrabPERF, GrabPERF.org, HTML, HTTP, IE, IM, it, Om, One, outage, performance, PHP, pr, search, Technorati, trim, web, Web Performance

I have reduced the number of measurements taken by the GrabPERF system down to 24 URLs every 15 minutes.

Also, I have purged the archives and data tables of all of the inactive measurements.

I feel a little sad, yet relieved that I am starting to trim this system back. I feel that unless I get some large outpouring of support, or interest, I will disable the system entirely on April 15, 2005.

The GrabPERF Measurement script is available here if you would like to take a crack at enhancing it.

Tags: data, GrabPERF, HTTP, IE, IM, IP, it, measurement, Om, performance, pr, trim, web, WebPerformance.Org