In the Customer Generation and Customer Retention articles of this series, the focus was on Web performance measurements designed to serve an audience outside of your organization. Starting with Business Operations, the focus shifts toward the use of Web performance measurements inside your organization.

Why Business Operations?

When I was initially developing these ideas with my colleague Jean Campbell, the idea was to call this section Reporting and Quality of Service. What we found was that this didn’t completely encompass all of the ideas that fall under these measurements. The question became: which part of the organization do reporting and QoS measurements serve?

What was clear was these were the metrics that reported on the health of the Web service to management and the company as a whole. This was the measurement data that the line of business tied to revenue and analytics data to get a true picture of the health of the online business.

What are you measuring?

Measurements for business operations need to capture the key metrics that are critical for making informed business decisions.

• How do we compare to our competitors?
• Are we close to breaching our SLAs?
• Are the third-parties we use close to breaching their SLAs?
• What parts of the site affect performance / user experience the most so we can set priorities?
• How does Web performance correlate with all the other data we use in our online business?

Every company will use different measures to capture this information, and correlate the data in different ways. The key is that you do use it to understand how Web performance ties into the line of business.

How often do I look at it?

Well, honestly, most people who work in business operations only need to examine Web performance once a day in a summary business KPI report (your company has a useful daily KPI report that everyone understands and uses, right?), and in greater detail at weekly and monthly management meetings.

The goal of the people examining business operations data is not to solve the technical problems that are being encountered, but to understand how the performance of their site affects the general business health of the company, and how it plays in the competitive marketplace.

What metrics do I need?

Business operations teams need to understand

• End-to-end response time for measured business processes
• Page-level response times for measured business processes
• Success rate of the transaction during the measurement period
• How third-parties are affecting performance
• How Web analytics and Web performance relate
• How different regions are affected by performance
• How does performance look from the customer ISPs and desktops

Detailed technical data is lost on these people. It is their role to take all of the data they have, and present a picture of the application as it affects the business, and discuss challenges that they face at a technical level in terms of how they affect the business.

Summary

For people who work at an extremely detailed level with Web measurement data (the topic for the next part of this series), Business Operations metrics seem light, fluffy, and often meaningless. But these metrics serve a distinct audience: the people who run the company. Frankly, if the senior business leaders at an organization are worried on a daily basis about the minute technical details taht go into troubleshooting and diagnosing performance issues, I would be concerned.

The objective of Business Operations measurements is to convey the health of the Web systems that support the business, and correlate that health with other KPIs used by the management team.

In the last article in this series, the concept of baselining your measurements was discussed. This is vital, in order for you and your organization to be able to identify the particular performance patterns associated with your site.

Now that’s under control, you’re done, right?

Not a chance. Remember that your site is not the only Web site your customers visit. So, how are you doing against all of those other sites?

Let’s take a simple example of the performance for one week for one of the search firms. This is simply an example; I am just too lazy to change the names to protect the innocent.

Doesn’t look too bad. An easily understood pattern of slower performance during peak business hours appears in the data, presenting a predictable pattern which would serve as a great baseline for any firm. However, this baseline lacks context. If anyone tries to use a graph like this, the next question you should ask is “So what?”.

What makes a graph like this interesting but useless? That’s easy: A baseline graph is only the first step in the information process. A graph of your performance tells you how your site is doing. There is, however, no indication of whether this performance trend is good or bad.

Examining the performance of the same firm within a competitive and comparative context, the predictable baseline performance still appears predictable, but not as good as it could be. The graph shows that most of the other firms in the same vertical, performing the identical search, over the same period of time, and from the same measurement locations, do not show the same daytime pattern of performance degradation.

The context provided by benchmarking now becomes a critical factor. By putting the site side-by-side with other sites delivering the same service, an organization can now question the traditional belief that the site is doing well because we can predict how it will behave.

A simple benchmark such as the one above forces a company to ask hard questions, and should lead to reflection and re-examination of what the predictable baseline really means. A benchmark result should always lead a company to ask if their performance is good enough, and if they want to get better, what will it take.

Benchmarking relies on the idea of a business process. The old approach to benchmarks only considered firms in the narrowly defined scope of the industry verticals; another approach considers company homepages without any context or reliable comparative structure in place to compensate for the differences between pages and sites.

It is not difficult to define a benchmark that allows for the comparison of a major bank to a major retailer, and a major social networking site, and a major online mail provider. By clearly defining a business process that these sites share (in this case let’s take the user-authentication process) you can compare companies across industry verticals.

This cross-discipline comparison is crucial. Your customers do this with your site every single day. They visit your site, and tens, maybe hundreds, of other sites every week. They don’t limit their comparison to sites in the same industry vertical; they perform cross-vertical business process critiques intuitively, and then share these results with others anecdotally.

In many cases, a cross-vertical performance comparison cannot be performed, as there are too many variables and differences to perform a head-to-head speed analysis. Luckily for the Web performance field, speed is only one metric that can be used for comparison. By stretching Web site performance analysis beyond speed, comparing sites with vastly different business processes and industries can be done in a way that treats all sites equally. The decade-long focus on speed and performance has allowed other metrics to be pushed aside.

Having a fast site is good. But that’s not all there is to Web performance. If you were to compare the state of Web performance benchmarking to the car-buying public, the industry has been stuck in the role of a power-hungry, horsepower-obsessed teenage boy for too long. Just as your automobile needs and requirements evolve (ok, maybe this doesn’t apply to everyone), so do your Web performance requirements.

Technorati Tags: Web+performance, benchmarking, performance+benchmarking, evolution

Just noticed that Bob Wyman of PubSub dropped in to look at the GrabPERF data we are collecting. [here]

Although Bob’s pride is evident, I do caution that being the top of a performance index is simply one measure of success. There is the whole discussion of relevancy of results, but that is beyond the scope of a performance-related discussion, such as the one I am encouraging on this blog.

Some of the concepts that I have been collaborating on have just been released to the public by my employer. Extending the outdated concepts of comparing Web sites simply on speed and availability, we have created Consistency and Optimization indices. [here]

These destroy any hope that faster is always better. Having a consistent Web performance and highly optimized site design counts for more as I get older than raw speed does.

My analogy is simple, and comes from the late, great Hunter S Thompson:

In horse breeding, for instance, there is a definite risk in breeding two fast horses who are both a little crazy. The offspring will likely be very fast and also very crazy. So the trick in breeding thoroughbreds is to retain the good traits and filter out the bad.

The Kentucky Derby is Decadent and Depraved

What good is a fast Web site that is “very crazy” and highly inconsistent?

Congrats to Bob Wyman and the PubSub team. I hope that your success continues.

Technorati: GrabPERF, Web Performance, Web Performance Measurement, Web Performance Monitoring, PubSub

IceRocket: GrabPERF, Web Performance, Web Performance Measurement, Web Performance Monitoring, PubSub

Back in November, I mentioned that I was working on the idea of new ways to benchmark the success of online businesses in today’s more mature operational environment. I am still working on the base ideas, but a colleague of mine has helped me coalesce some ideas, and they are now forming the foundation of the concepts my company will begin using internally to more deeply understand the various Web performance benchmarks we monitor.

For those who use the existing Web performance benchmarks to determine the success and failure of your online business, you understand how thin the veneer is on these benchmarks. They do not provide true insight into the operational success of an online business, and they are more likely to sow the seeds of distrust between IT and Business operations in the long-term by creating an artificial standard which becomes the goal.

If an online business truly wants to achieve and maintain exemplary Web performance numbers, it has to start with a strong foundation, and build on it. Why? The team I work with spends a lot of time trying to understand and reverse engineer the broken processes, designs, and architectures that were laid out in order to get big fast. After 3-4 years of technical starvation and underfunding, these online businesses are beginning to show strain; the temporary fix has become the permanently broken process.

The rush into the Web analytics space in the last few weeks is a key sign that companies now see value in and want to exploit the vast quantities of data that they collect on their traffic daily. Web analytics is an astoundingly complex field, but most people boil it up to a single concept: How many Unique Page Views did I get?

Unique Page Views is an outdated Web server analytics metric. It does not tell me anything about the business, other than it has a lot of traffic. Back in the “eyeballs are everything” period, this would have been a big deal. Now, I say so what, and start asking:

• From where
• Dialup? Broadband?
• How many were able to successfully complete their transactions?
• What paths are most visited
• Average spend by connection type?
• Average spend by hour?
• etc.

Like Unique Page Views, the average Web performance and availability of a Web page or transaction does not accurately represent the overall health of any online business. Within the large populations of data that exist at the Web measurement firms, there is a wealth of data that could be used to clearly expose more important benchmarking statistics.

If you are from an online business, you already understand that the average performance over an artificially-defined period of time is a very inaccurate way to measure the success of the online business. However, it is the accepted standard in the field. Underlying those aggregated values, there are clearly-defined statistical methods which can be used to extract even more meaningful information from the mass of measurement data.

I would discuss more of the ideas and concepts that we are working on, but I know that I do get visitors from our competitors, so I will have to keep our ideas under wraps for right now.

But I want to hear yours. What does your online business use as a benchmark for success? Standard avergae performance and availability? Or something more complex that examines the performance data as a complete population, as opposed to an aggregated summary value? Does your firm tie business goals and objectives into the performance benchmarks so that people across the company can understand how the business is succeeding, and how delivering a good, bad, and downright awful online performance experience can affect the bottom line?

This is an exciting time to have access to large amounts of data on the health of the Internet.

Ouch. I thought this one was worthy of some analysis. Google acquire Urchin. [and here]

Why does this make sense? As one of the comments to the link above states, the link between search algorithms makes some sense, as you will be able quickly isolate and identify trends within the reams of visitor data that is collected daily.

This makes linking business performance and visitor traffic that much easier for large enterprise firms. If they can link visitor traffic quickly to revenue and cost data with a simple and powerful search tool, the sale becomes that much easier.

I am working on a project to re-define Web performance benchmarks, and this type of tool would be crucial in that process. Web Benchmarking as it is currently stands is broken and outdated. Attempts are being made on all sides to try and re-define how businesses measure and benchmark their Web performance so that it can fit into a larger business context.

Is a faster site more profitable? Or simply more costly? Does increased traffic improve or reduce revenue?

A tool that allow companies to quickly link performance and analytics data gets firms that much closer to a holistic view of Web performance.

Remember how Comcast Re-IP’ed the entire neighbourhood this week? Well, I can measure the Sears hompage again. For those who don’t know the story, you can read more here.

The ethical question that I face is whether I should or not because I can. In the Web performance business, the companies that I work for have always held that if the site is public, then it can be measured, even if the site owner is not the one requesting the measurement (competitive benchmarking, public performance indices, etc.).

I am not going to burn the bridge with Sears at this point, but does anyone else have an opinion? How would your company react if it found itself being measure/monitored without it’s knowledge/consent?

Web performance benchmarks have been a part of the industry since the beginning. However, it is not 1999 anymore. What do these benchmarks really mean?

Simple, aggregated performance and success rate values based on a limited dataset for a finite time period give a very tiny perspective into the world of Web performance. The operation of a multi-billion dollar enterprise should not hinge on the ranking a firm receives in these comparative exercises.

The question I pose to the Web performance industry is this: If you could walk away from the heritage of benchmarking that we have inherited, what would you replace it with? How could you justify your IT spend on Web infrastructure to your business leaders? How could you demonstrate that your Web infrastructure was a benefit and not a drain to the business?

I don’t have the silver bullet to these questions, but I do have some very primitive new ideas running around in my head, trying to evolve. I would love to here what other Web performance IT and business leaders have to say on this concept.