While this post is aimed at Web performance, the curse of the single metric affects our everyday lives in ways that we have become oblivious to.

When you listen to a business report, the stock market indices are an aggregated metric used to represent the performance of a set group of stocks.

When you read about economic indicators, these values are the aggregated representations of complex populations of data, collected from around the country, or the world.

Sport scores are the final tally of an event, but they may not always represent how well each team performed during the match.

The problem with single metrics lies in their simplicity. When a single metric is created, it usually attempts to factor in all of the possible and relevant data to produce an aggregated value that can represent a whole population of results.

These single metrics are then portrayed as a complete representation of this complex calculation. The presentation of this single metric is usually done in such a way that their compelling simplicity is accepted as the truth, rather than as a representation of a truth.

In the area of Web performance, organizations have fallen prey to this need for the compelling single metric. The need to represent a very complex process in terms that can be quickly absorbed and understand by as large a group of people as possible.

The single metrics most commonly found in the Web performance management field are performance (end-to-end response time of the tested business process) and availability (success rate of the tested business process). These numbers are then merged and transformed by data from a number of sources (external measurements, hit counts, conversions, internal server metrics, packet loss), and this information is bubbled up in an organization. By the time senior management and decision-makers receive the Web performance results, that are likely several steps removed from the raw measurement data.

An executive will tell you that information is a blessing, but only when it speeds, rather than hinders, the decision-making process. A Web performance consultant (such as myself) will tell that basing your decisions on a single metric that has been created out of a complex population of data is madness.

So, where does the middle-ground lie between the data wonks and the senior leaders? The rest of this post is dedicated to introducing a few of the metrics that will, in a small subset of metrics, give a senior leaders better information to work from when deciding what to do next.

A great place to start this process is to examine the percentile distribution of measurement results. Percentiles are known to anyone who has children. After a visit to the pediatrician, someone will likely state that “My son/daughter is in the XXth percentile of his/her age group for height/weight/tantrums/etc”. This means that XX% of the population of children that age, as recorded by pediatricians, report values at or below the same value for this same metric.

Percentiles are great for a population of results like Web performance measurement data. Using only a small set of values, anyone can quickly see how many visitors to a site could be experiencing poor performance.

If at the median (50th percentile), the measured business process is 3.0 seconds, this means that 50% of all of the measurements looked at are being completed in 3.0 seconds or less.

If the executive then looks up to the 90th percentile and sees that it’s at 16.0 seconds, it can be quickly determined that something very bad has happened to affect the response times collected for the 40% of the population between these two points. Immediately, everyone knows that for some reason, an unacceptable number of visitors are likely experiencing degraded and unpredictable performance when they visit the site.

A suggestion for enhancing averages with percentiles is to use the 90th percentile value as a trim ceiling for the average. Then side-by-side comparisons of the untrimmed and trimmed averages can be compared. For sites with a larger number of response time outliers, the average will decrease dramatically when it is trimmed, while sites with more consistent measurement results will find their average response time is similar with and without the trimmed data.

It is also critical to examine the application’s response times and success rates throughout defined business cycles. A single response time or success rate value eliminates

• variations by time of day
• variations by day of week
• variations by month
• variations caused by advertising and marketing

An average is just an average. If at peak buiness hours, response times are 5.0 seconds slower than the average, then the average is meaningless, as business is being lost to poor performance which has been lost in the focus on the single metric.

All of these items have also fallen prey to their own curse of the single metric. All of the items discussed above aggregate the response time of the business process into a single metric. The process of purchasing items online is broken down into discrete steps, and different parts of this process likely take longer than others. And one step beyond the discrete steps are the objects and data that appear to the customer during these steps.

It is critical to isolate the performance for each step of the process to find the bottlenecks to performance. Then the components in those steps that cause the greatest response time or success rate degradations must be identified and targeted for performance improvement initiatives. If there are one or two poorly performing steps in a business process, focusing performance improvement efforts on these is critical, otherwise precious resources are being wasted in trying to fix parts of the application that are working well.

In summary, a single metric provides a sense of false confidence, the sense that the application can be counted on to deliver response times and success rates that are nearly the same as those simple, single metrics.

The average provides a middle ground, a line that says that is the approximate mid-point of the measurement population. There are measurements above and below this average, and you have to plan around the peaks and valleys, not the open plains. It is critical never to fall victim to the attractive charms that come with the curse of the single metric.

Kevin Tillman, the brother of the late Pat Tillman, has written an essay that will make it hard for anyone to justify Iraq.

How can the Rumsfeld, Rice, Cheney, Rove, and Bush argue their way out of this essay?

Remember, the Gettysburg Address took less than 2 minutes to deliver.

Pass it on.

His comrades fought beside him, Van Owen and the rest
But of all the Thompson gunners, Roland was the best
So the CIA decided they wanted Roland dead
That son of a bitch Van Owen blew off Roland’s head

There is always something something soothing in the words of Warren Zevon, even when they describe the wandering ghost of a Biafran War veteran. They make the human condition, well, more human.

And the human condition is an important thing to remember on a Monday morning. With headlines shrieking of suffering, you have to sit back and wonder why; why are we still here?

I called up my friend LeRoy on the phone
I said, Buddy, I’m afraid to be alone
‘Cause I got some weird ideas in my head
About things to do in Denver when you’re dead

The virus I have in my system right now has a greater chance of surviving into the next century than the human race does. And sometimes I wonder if Zevon and Gunter Thompson always knew that, and used this knowledge to fuel a life of what we see as madness. In fact, their madness has a stronger truth than the artifical and virtual lives so many of us (including your author) lead.

It’s not melancholy that drives this rambling rant; it is a smiling realization that we are all the same, and in the end, for 99.99% of us, life is pretty darn empty in the end.

Well, I pawned my Smith Corona
And I went to meet my man
He hangs out down on Alvarado Street
By the Pioneer chicken stand

Carmelita hold me tighter
I think I’m sinking down
And I’m all strung out on heroin
On the outskirts of town

The words of a strung out junkie can ring more true in this world than the pronouncements of great people. The battles we, the 99.99% unwashed masses, face everyday are what keep societies working. Without our willingness to get up and battle every day, the great people would drown in a wave of chaos that makes today’s world seem like a tea party.

Viruses have it easy. The dead have it easy. It’s the living who stare down that dark tunnel every morning, and walk toward the deeping darkness (humans who say they see the light are more deluded than the rest of us).

Embrace viruses; embrace pain; embrace love; embrace every breath you take.

In the end, that’s what you take with you.

Left eye, right eye
Take a look around
Everybody’s heading
For a hole in the ground
And it’s the Dance of Shiva
It’s the Twilight of the Gods
Thunder and lightning
‘Til the break of dawn

Monkey wash donkey rinse
Going to a party in the center of the earth
Monkey wash donkey rinse
Honey, don’t you want to go?

Technorati Tags: Warren Zevon, Zevon, virus, life, death, philosophy, existentialism

Heh.

Count on SkyNews to bring you an unintended laugh.

Via: Fred Wilson

Then do the following:

Turns out that if you type “failure” into Google and hit “I feel lucky” you’ll get the official biography of, well, I don’t want to ruin the suprise.

Heh!

Via: Scoble

Via GapingVoid

Been there; have the scars.

Hey Karl, can you spin this?

If you are the CEO and your underling might have committed a serious crime… would you ask them? Either Rove told Bush the truth and Bush doesn’t care, Rove lied to the President and the President can forgive him, or the President doesn’t care and hasn’t asked. None of those reveal good qualities in our Commander in Chief. This is going to burn him one way or another in the end.

Love it!

Via Daily Bubble

Technorati: Karl Rove, KarlRove, White House

This post has been slanted by an article I read today about Yankee Group analyst Laura Didio (hmmm…no bio on the site), and an encounter I had today with a real industry analyst. [James Governor points to this article as well.]

Ms. Didio has been accused of placing a very hard slant towards Microsoft in most of her Operating System analyses, to the point of being almost completely invalid and useless. If Microsoft pays the bill, I don’t have a problem with her coming out and indicating where Microsoft server OSes are strong compared to Linux. But to take facts which are inconclusive and then skewing them to favour the client…well, please get out of my office.

Then I was a passive participant in a call with a well-known analyst (no name or firm here). My takeaway frm the call was: I want his job. Not because he had brilliant things to say, or incredible insights to offer, but tbecause he was being paid repulsive sums of money to state the obvious.

I spent most of the meeting shaking my head and wondering how he did it. It was like delivering a monlogue in an echo chamber: he had one thing to say, and anything that our team brought up was routed back to his topic, in a cursory way.

It was clear he had no idea what our company does, what our positioning and strategy are, and how our services could help our customers.

And we paid for this.

The analyst industry is so corrupt and meaningless. I am glad that there are folks like ARmageddon, GartnerWatch, and Analyst Insight out there to expose the industry.

I hate blogging about blogging, but the whole area of analyst research is being eroded and corroded by blogs. Companies are doing their own research using Technorati and Feedster and making their own judgements.

The best way to make analysts extinct is for companies to tell their own story, in their own words, within their own context, and give it meaning.

Analysts have stolen our the ability to find and tell our own stories.

As soon as the characters started speaking in this vicious and detailed bit of parody, I knew it was Canadian. [here, 32mb MP4]

And sure enough, Avion Films is a Canadian ad agency.

I miss my home country.

care of Site-9.

Hence the need to target those frivolous Asbestos Lawsuits…

W.R. Grace indicted over asbestos claims.

I mentioned this a while back…some people close to The Obsfucator are tied to this asbestos thing.

I have no sympathy for the asbestos industry. A high-school friend of mine lost his father due to a one-time, one-day exposure to asbestos 30 years ago. This material is hyper-dangerous.

It killed Warren Zevon. [here] ‘Nuff said.

Seth:

Typical notes about how we don’t know each other, blah, blah, blah.

On point, your comment on the PC. You said:

I had to use a PC today in order to run Exact Target to do a mailing. I was stunned and astonished at how much the experience has degraded since my last exposure.

I know what you mean. But I am stuck.

I could mortgage myself and use a Mac. I would kill for a Powerbook in any size. I think they are cool. I love Apple.

Truth is, I work for Windows Regime. I have a mortgage and two kids. My disposable income does not exist.

I would love to say that I have tried the alternative, but Linux Desktops don’t cut it for me. I want a machine that is dead sexy. I want an OS that gets it.

I want a Mac. And I can’t have one.

I can have Firefox on Windows, but that’s just not the same.

Thanks for the Firefox Evangelizing…and for reminding that the OS I use isn’t sexy.

Sorry Scoble.

Stephen