While this post is aimed at Web performance, the curse of the single metric affects our everyday lives in ways that we have become oblivious to.

When you listen to a business report, the stock market indices are an aggregated metric used to represent the performance of a set group of stocks.

When you read about economic indicators, these values are the aggregated representations of complex populations of data, collected from around the country, or the world.

Sport scores are the final tally of an event, but they may not always represent how well each team performed during the match.

The problem with single metrics lies in their simplicity. When a single metric is created, it usually attempts to factor in all of the possible and relevant data to produce an aggregated value that can represent a whole population of results.

These single metrics are then portrayed as a complete representation of this complex calculation. The presentation of this single metric is usually done in such a way that their compelling simplicity is accepted as the truth, rather than as a representation of a truth.

In the area of Web performance, organizations have fallen prey to this need for the compelling single metric. The need to represent a very complex process in terms that can be quickly absorbed and understand by as large a group of people as possible.

The single metrics most commonly found in the Web performance management field are performance (end-to-end response time of the tested business process) and availability (success rate of the tested business process). These numbers are then merged and transformed by data from a number of sources (external measurements, hit counts, conversions, internal server metrics, packet loss), and this information is bubbled up in an organization. By the time senior management and decision-makers receive the Web performance results, that are likely several steps removed from the raw measurement data.

An executive will tell you that information is a blessing, but only when it speeds, rather than hinders, the decision-making process. A Web performance consultant (such as myself) will tell that basing your decisions on a single metric that has been created out of a complex population of data is madness.

So, where does the middle-ground lie between the data wonks and the senior leaders? The rest of this post is dedicated to introducing a few of the metrics that will, in a small subset of metrics, give a senior leaders better information to work from when deciding what to do next.

A great place to start this process is to examine the percentile distribution of measurement results. Percentiles are known to anyone who has children. After a visit to the pediatrician, someone will likely state that “My son/daughter is in the XXth percentile of his/her age group for height/weight/tantrums/etc”. This means that XX% of the population of children that age, as recorded by pediatricians, report values at or below the same value for this same metric.

Percentiles are great for a population of results like Web performance measurement data. Using only a small set of values, anyone can quickly see how many visitors to a site could be experiencing poor performance.

If at the median (50th percentile), the measured business process is 3.0 seconds, this means that 50% of all of the measurements looked at are being completed in 3.0 seconds or less.

If the executive then looks up to the 90th percentile and sees that it’s at 16.0 seconds, it can be quickly determined that something very bad has happened to affect the response times collected for the 40% of the population between these two points. Immediately, everyone knows that for some reason, an unacceptable number of visitors are likely experiencing degraded and unpredictable performance when they visit the site.

A suggestion for enhancing averages with percentiles is to use the 90th percentile value as a trim ceiling for the average. Then side-by-side comparisons of the untrimmed and trimmed averages can be compared. For sites with a larger number of response time outliers, the average will decrease dramatically when it is trimmed, while sites with more consistent measurement results will find their average response time is similar with and without the trimmed data.

It is also critical to examine the application’s response times and success rates throughout defined business cycles. A single response time or success rate value eliminates

• variations by time of day
• variations by day of week
• variations by month
• variations caused by advertising and marketing

An average is just an average. If at peak buiness hours, response times are 5.0 seconds slower than the average, then the average is meaningless, as business is being lost to poor performance which has been lost in the focus on the single metric.

All of these items have also fallen prey to their own curse of the single metric. All of the items discussed above aggregate the response time of the business process into a single metric. The process of purchasing items online is broken down into discrete steps, and different parts of this process likely take longer than others. And one step beyond the discrete steps are the objects and data that appear to the customer during these steps.

It is critical to isolate the performance for each step of the process to find the bottlenecks to performance. Then the components in those steps that cause the greatest response time or success rate degradations must be identified and targeted for performance improvement initiatives. If there are one or two poorly performing steps in a business process, focusing performance improvement efforts on these is critical, otherwise precious resources are being wasted in trying to fix parts of the application that are working well.

In summary, a single metric provides a sense of false confidence, the sense that the application can be counted on to deliver response times and success rates that are nearly the same as those simple, single metrics.

The average provides a middle ground, a line that says that is the approximate mid-point of the measurement population. There are measurements above and below this average, and you have to plan around the peaks and valleys, not the open plains. It is critical never to fall victim to the attractive charms that come with the curse of the single metric.

Tags: business process, components, curse, cycles, metrics, percentiles, performance, statistics, steps, trim, truth, variation, web, Web Performance, Web performance management, Web performance management field, Web performance measurement data, Web performance results

Some of you may know of this frightening shot of Doc Searls mouth.

Well, move over Doc! I just learned at the dentist this morning that my bridge has to come out, as the foundation teeth that hold it in place (it’s a permanent plate) are…well, failing is the polite way to say it.

I have had the bridge for 20 years, so this is is no surprise to me, and one of the reasons that I have avoided seeing a dentist for so long.

I also learned that it could cost up to $25,000 to have the cadillac of replacements.

Or $10,000 if I want a job that will have to be fixed in another 10 years.

Or $1,000 if I don’t mind feeling like my grandparents, putting my teeth in a glass by the bed every night.

Well, you guessed it: damn the self-image! I am going for the removable partial plate.

And how did I get into this mess in the first place? It’s simple: I ran a complex, real-time experiement that clearly demonstrated the force required to remove teeth. The components included:

• One bicycle
• One truck
• One asphalt roadway

In order to complete the experiment, the bicycle had to precisely clip the bumper of the truck. This was difficult to perform, as the bicycle was moving at high speed and the truck (which would not be able to see the bicycle until the last minute) had to swerve very precisely in front of the bicycle…and rider.

The MythBusters would have been proud.

So, unlike many Canadian men (and women), I lost my teeth in a cycling accident, not as a result of an on-ice altercation or an attempt to be Gump Worsley.

So, if you meet me in the future, ask me to pop out my teeth.

I might just do it.

Technorati Tags: dentist, dental plate, dental bridge, doc searls, dentures, gump worsley, hockey

Tags: ASP, components, HTML, HTTP, IE, IM, IP, it, Om, One, pr, Technorati

I noticed that a new version (4.1.12) of MySQL was up on their site last night. I grabbed the RPMs (Yes, I am a binary-loving weenie, not a hardcore source jockey) and installed them.

For those of you who install MySQL from RPMs know that it takes 4 packages to get all of the components up and running correctly. I got 3 of the 4 running no problem.

The one that bombed is the one that contains the main server binaries. All sorts of backtrace and coredump type errors, and then no response from the DB. So I re-installed the previous Server RPM, and I am up and running. I just figured I am an idiot and moved on with my evening.

This morning, I went to the MySQL site. Lo and behold, all of the 4.1.12 downloads have been pulled.

I don’t feel like such an idiot anymore. And I am not alone: here are the Severity 1 bugs for MySQL 4.1.12. Two of them are identical to what I was seeing.

Tags: ads, bugs, components, HTTP, IE, IM, ISP, it, mysql, Om, One, PHP, pr, run, search, server

I submitted a presentation proposal for OSCON 2005 just now. The abstract is below.

The Open Source community has driven the online world for the last decade. PHP, PERL, Apache, Java, and MySQL are all major components of large online enterprises.

However, putting an application online and ensuring that it satisfies the performance, availability and reliability demands of the increasingly knowledgeable online consumer are often two separate concerns.

Performance should not be an afterthought; performance should be a leading force in creating a Web application.

Using simple Open Source Tools, Web performance measurement solutions can be built that rival commercial solutions. But what does this data tell you? And how do you turn this into useful business information?

This discussion will expose the participants to key Web performance metrics that make sense to both technology and business leaders in your organization.

I have a snowball’s chance in hell of having it accepted, as it is not hip, technical or trendy, and I am not an Open Source Guru, but if you design stuff for the Web, then you better be ready to have your site examined in detail, because if you don’t do it, your customers will.

Tags: apache, app, application, business, community, components, customers, data, design, EAD, HTTP, IE, IM, information, IP, it, measurement, metrics, mysql, Om, One, online, online world, OReilly, performance, performance measurement, Perl, PHP, pr, Reliability, web, web app, web application, Web Performance, Web performance measurement

Despite the doom-and-gloom title to this post, I fully sympatize with what the FBI is trying to accomplish, and the very public pain that they will feel as a result of the failure of Virtual Case File 1.0.

What I find interesting is that the biggest technology company no one has ever heard of, SAIC, failed to deliver a product that met the client’s needs. They are noted for their cutting-edge development and technology, especially in skunkworks and black ops.

This project demonstrates that incredibly complex software projects can and will fail to deliver unless the needs of the users are carefully considered, and the project is not rushed for political purposes.

What is the goal of Virtual Case File? I hypothesize that it is to make the process of handling case documentation and correlation more effective and efficient, with the additional benefit of being able to link disparate bits of information into a more effective whole more quickly, saving lives, etc.

Ever line of code needs to be developed with this purpose in mind. Does this function allow a user to achieve the goals that have been defined for the entire project? If not, what do we need to do to improve this function?

It is very easy to lose sight of the corporate goal when working on atomic-level components of a larger project. Happens to me every day. That is why I sit back at the end of each day and consider how my efforts contributed to my strategic personal and professional goals. I have to do it, or else I get trapped by the minutiae.

I wish the FBI luck, as this project is a much-needed advance for law-enforcement.

Tags: app, client, components, corporate, government, HTTP, IE, IM, information, ISP, it, live, Om, One, personal, pr, users