Measure what is measurable, and make measurable what is not so.

Galileo Galilei

The greatest challenge facing companies today is not finding ways to measure performance. The key issue is one of understanding what should be measured and validating that there is agreement on what the purpose of the measurement is.

Organizations are complex. And with complexity arises the need to gather data for different purposes. In my series discussing Why Web Measurements?, I broke organizations down into four groups, each one having distinctly different needs for measurements and data. While this series focuses on Web performance, the four categories (Customer Generation, Customer Retention, Business Operations, and Technical Operations) can be broadly applied to all aspects of your business.

In each of the four categories, whether it is for Web performance or financial analysis, determining what and why to measure is a critical predecessor to the establishment of measurements and the examination of data.

2009 will be a year of reflection and retrenchment. Companies will be examining all aspects of their business, all of their relationships with vendors, all of the ways they measure themselves. The question that must be asked before succumbing to the rushing panic of cost-cutting and layoffs is: Do you fundamentally understand why and what you measure and what it is really telling you?

Give almost any human being a hammer, and they will know what to do with it. Modern city dwellers, ancient jungle tribes, and most primates would all look at a hammer and understand instinctively what it does. They would know it is a tool to hit other things with. They may not grasp some of the subtleties, such as that is designed to drive nails into other things and not beat other creatures into submission, but they would know that this is a tool that is a step up from the rock or the tree branch.

Simple tools produce simple results. This is the foundation of a substantial portion of the Software-as-a-Service (SaaS) model. SaaS is a model which allows companies to provide a simple tool in a simple way to lower the cost of the service to everyone.

Web performance data is not simple. Gathering the appropriate data can be as complex as the Web site being measured. The design and infrastructure that supports a SaaS site is usually far more complex than the service it presents to the customer. A service that measures the complexity of your site will likely not provide data that is easy to digest and turn into useful information.

As any organization who has purchased a Web performance measurement service, a monitoring tool, a corporate dashboard expecting instant solutions will tell you, there are no easy solutions. These tools are the hammer and just having a hammer does not mean you can build a house, or craft fine furniture.

In my experience, there are very few organizations that can craft a deep understanding of their own Web performance from the tools they have at their fingertips. And the Web performance data they collect about their own site is about as useful to them as a hammer is to a snake.

There is clear dissatisfaction with the current state of marketing among the social media mavens.

• Fred Wilson and Union Square Ventures are looking for companies to invest in to take advantage of this.
• BuzzLogic releases their conversational ad service.
• The Inquisitr moves from AdSense to Technorati Media, indicating a potential shift at b5 Media.
• Lookery is providing demographic information that is relevant to social media providers and shaping ad presentation that way
• A completely off-the-radar person such as myself realizes that there is gold in what holds the attention, and that’s not always the latest and greatest

So what can be done? Jeff Jarvis points out that the problem lies with measurement. I agree, as there is only value in a system where all of the people involved agree on what the metric of record will be, and how it can be validly captured.

Currently CPM is the agreed upon metric. In a feed based online world, how does a CPM model work? And, most importantly, why would I continue to place your ads on my site if all your doing is advertising to people based on the words on the page, rather than who is looking at the page and how often that page is looked at.

In effect, advertisers should be the ones thrying to figure out how to get into the community, get into the conversation. As an advertiser, don’t you want to be where the action is? But how do you find an engaged audience in an online world that makes a sand castle on the beach in a hurricane look stable?

The challenge for advertisers is to be able to find the active communities and conversations effectively. The challenge for content creators and communities is to understand the value of their conversations, the interactions that people who visit the site have with the content.

In effect, a social media advertising model turns the current model on its head. Site owners and community creators gain the benefit of being attractive to advertisers because of the community, not because of the content. And site owners who understand who visits their site, what content most engages them, how they interact with the system will be able to reap the greatest rewards by selling their community as a marketable entity.

And Steven Hodson rounds out the week’s think on communities by throwing out the subversive idea that communities are not always free (as in ‘beer’, not as in ‘land of’). If a community has paid for the privilege of coming together to participate in communal events and discussions, then can’t that become an area for site owners to further control the cost of advertising on their site?

While the benefit of reduced or no marketing content is the benefit of many for-pay communities, this benefit can be used by site owners by saying that an advertiser can have access to the for-pay community at the cost of higher ad rates and smaller ads. The free community is a completely different set of rules, but there are also areas in the free community that are of higher value than others.

In summary, the current model is broken. But there is no way to measure the value of a Twitter stream, a FriendFeed conversation, a Disqus thread, or a Digg rampage. And until there is, we are stuck with an ad model that based on the words on the page, and not the community that created the words.

I know that I have been bad at blogging news about GrabPERF, but today there is some. In the last two weeks, we have added two measurement locations: Washington DC AOL and Argentina LaNacion.

Thanks to Carson Evans of AOL, and Jose Falvo and Leonardo Lancellotta of LaNacion for helping out with the installation process.

Today’s biggest victim of Black Friday appears to be Sears

Sears measurement data for the last 8 hours can be found here.

UPDATE: It gets worse for Sears.

Technorati Tags:
Grabperf, Sears, Black Friday, Web Performance

mon.itor.us, a service which also provides free Web performance measurement services, appears to be having a wee problem.

The most recent GrabPERF data on this site is available here. The issue may be corrected by the time you look at the data.

I don’t wish suffering like this on anyone. GrabPERF had it’s own 3-4 day outage a few months ago. It’s just sad to see when monitoring services go down.

Part of the problem with making a lot of East to West trips across the US is that the flights back are a nightmare for timing. You either lose most of a day or have to take a red-eye.

Well, taking the red-eye has only one advantage: you get to camp in the airline lounge and get some work done that you have been too tired/jetlagged to deal with.

I am in the United Red Carpet Lounge. It is 18:33 PDT, and my flight boards at 22:00 PDT. I have been here since 15:00 PDT. I have caught up with a client project, completed my (dreaded) weekly timesheet, booked accommodation for my Columbus, OH trip, and tidied up an analysis script that I use to process client measurement data.

Seems odd that this is the most productive time of the week.

This trip is a 48-hour turnaround from Boston to LA to finish up the project for a large client. The first half was handled by some colleagues, and was detailed here.

Then,when I get home, I have to go to NYC (Long Island City actually) for the day on Monday. And, as I mentioned above, Columbus which is set for the middle of the month.

Compared to some jet-setters out there, this is nothing. But I travelled so much between August 15 and November 15 that I went from 2,000 miles on United to well into Premier (24,000+ miles). This trip makes the fourth cross-country trip in 2.5 months. Not bad for a homebody.

I have three more hours. Think I will sit back and watch Trois Couleurs: Blanc.

Had an on-site meeting with Dorion Carroll and the Technorati team. We discussed some ways to extend and expand the GrabPERF infrastructure, and improve the measurement gathering.

Look for changes in the next 6 months that could radically improve the system you know and love!

It has been at least a year since I last updated everyone on the state of GrabPERF. That’s because for most of the last year, the system has been rolling along without a hitch or a major systemic change. The last major change to the agent code was alomost exactly a year ago, when I added the ability to capture text matches.

It wasn’t until last week, however, that I allowed folks to be able to see the results of these text match failures. Let’s just say that motivation has been low and my real job has been keeping me busy.

I did want to share the growth, and mellowing of the system as it progresses into year three.

Total Measurements Per Day

Unique Tests Measured Per Day

Back in July 2005 when I started this grand experiment, I was gathering 10,000 measurements a day from less than forty tests. The system spiked at 390,000 measurements per day (April 2006) and 147 tests (June 2006).

Starting just after that, I started reducing the number of tests to improve system efficiency, and began developing the text match capability.

There have been some changes to the number of measurements, but on the whole, the system has been completely stable for the last 12 months.

As some of you may have noted, I have add some new features in the last 10 days, and re-organized the structure of the system to allow for better tracking of usage. Over the next few months, I will be attacking the code to make it process things more efficiently, but not substantially change the appearance or functionality of GrabPERF.

It has been noted by some commentators that the design doesn’t pop and sizzle. No AJAX, DHTML, or other flashy gizmos. Guess what? The system is designed to deliver data efficiently and effectively. And as someone who has seen the performance fall-out from badly delivered Web 2.0 implementations, I will stick with clunky and effective, as, in the end, you gotta put all those bytes on the wire.

For those that have stuck with the system over the last year, thanks. I enjoy delivering the best measurement data money can’t buy, and hope you stick around for the ride.

Last night, I got motivated.

Ok, I got manic. Goes with my life.

As a part of that mania, I had a breakthrough on how to present GrabPERF data that I’ve actually been collecting for nearly a year: text match failures.

GrabPERF has the ability to match text on page results using a standard PERL regex. By putting a regex into the measurement configuration, I can confirm that the data downloaded matches what should be there.

If there is not a match, the headers and page text are captured and inserted into a table. Up until yesterday, I was the only one who could view the data. Now, if you go to the graph configuration page (http://grabperf.org/measure_page.php?test=[insert test id here]), and see the following type of result, then click through the links.

If your graph configuration page says no text match configured, and you want one, let me know!