Measure what is measurable, and make measurable what is not so.

Galileo Galilei

The greatest challenge facing companies today is not finding ways to measure performance. The key issue is one of understanding what should be measured and validating that there is agreement on what the purpose of the measurement is.

Organizations are complex. And with complexity arises the need to gather data for different purposes. In my series discussing Why Web Measurements?, I broke organizations down into four groups, each one having distinctly different needs for measurements and data. While this series focuses on Web performance, the four categories (Customer Generation, Customer Retention, Business Operations, and Technical Operations) can be broadly applied to all aspects of your business.

In each of the four categories, whether it is for Web performance or financial analysis, determining what and why to measure is a critical predecessor to the establishment of measurements and the examination of data.

2009 will be a year of reflection and retrenchment. Companies will be examining all aspects of their business, all of their relationships with vendors, all of the ways they measure themselves. The question that must be asked before succumbing to the rushing panic of cost-cutting and layoffs is: Do you fundamentally understand why and what you measure and what it is really telling you?

In my life as a consultant, I often discuss What Web performance data means and how to interpret it to solve problems. Solving the problems is, however, inherently based on whether the data that is collected is meaningful. In trying to find data that is meaningful, we have found that there are four categories that Web performance measurements fall into: Customer Generation, Customer Retention, Business Operations, and Technical Operations.

Customer Generation

How can you use Web performance measurement data to outperform your competition and impress your prospects. Read it here!

Customer Retention

Impress your customers with your skill and responsiveness, and keep the competition from sneaking in the back door. Read it here!

Business Operations

Know how you are doing against your competition and prioritize what you need to do to stay ahead. Read it here!

Technical Operations

Know what to measure and how often to keep a detailed eye on your internal systems and external performance. Read it here!

In the first three parts of this series, the focus has been on the business side of the business: Customer Generation, Customer Retention, and Business Operations. The final component of any discussion of why companies measure their Web performance falls down to Technical Operations.

Why is Technical Operations last?

This part of the conversation is the last, mainly because it is the most mature. A technical audience will understand the basics of a distributed Web performance measurement system, or a Web analytics system, or a QA testing tool without too much explanation. The problems that these tools solve are well-defined and have been around for many years.

Quickly thinking about these types of problems makes it clear, however, that the kind of data needed in a technical operations environment is substantially different than that which is needed at the Business Operations level. Here, the devil is in the details; at Business Operations, the devil is in the patterns and trends.

What are you trying to measure?

The short answer is that a Technical Operations team is trying to measure everything. More data is better data at this level. The key is the ability to correlate multiple sources of system inputs (Web performance data, systems data, network data, traffic data, database queries, etc.) to detect the patterns of behavior which could indicate impending crises or complete system outage, or simply a slower than expected response time during peak business hours.

And while Technical Operations teams thrive on data, they do not thrive on explaining this data very well to others. So the metrics which are important in one organization may not be the key ones in another. Or they may be called by a completely different name. Which is why Technical Operations sigh and throw up their hands in despair when talking to management who are working from Business Operations data.

How do you measure it?

Measure early. Measure often.

This sums up the philosophy of most Technical Operations teams. They want to gather as much data as possible. So much data that the gathering of this data is often one step away from affecting the performance of their own systems. This is how the scientific mind works. So, be prepared to control this urge to measure and instrument everything with a need to ensure that the system is operationally sound.

Summary

Even in the well-developed area of Technical Operations, there is still opportunity to ensure that you are measuring the right things the right way. Do an audit of your measurements. Ask the question “why do we measure this this way?”.

Measure meaningful things in a meaningful way.