David Cancel and I have had sort of a passing vague, same space and thought process, living in the same Metropolitan area kind of distant acquaintance for about the same year.

About 2-3 months ago, he wrote a pair of articles discussing the efforts he has undertaken in order to try and offload some of the traffic to the servers for his new company, Lookery. While they are not current, in the sense that time moves in one direction for most technical people, and is compressed into the events of the past eight hours and the next 30 minutes, these articles provide an insight that should not be missed.

These two articles show how easily a growing company that is trying to improve performance and customer experience can achieve measureable results on a budget that consists of can recycling money and green stamps.

Measuring your CDN

A service that relies on the request and downloading of a single file from a single location very quickly realizes the limitations that this model imposes as traffic begins to broaden and increase. Geographically diverse users begin to notice performance delays as they attempt to reach a single, geographically-specific server. And the hosting location, even one as large as Amazon S3, can begin to serve as the bottleneck to success.

David’s first article examines the solution path that Lookery chose, which was moving the tag, which drives the entire opportunity for success in their business model, onto a CDN. With a somewhat enigmatic title (Using Amazon S3 as a CDN?), he describes how the Lookery team measured the distributed performance of their JS tag using a free measurement service (not GrabPERF) and compared various CDNs against the origin configuration that is based on the Amazon S3 environment.

This deceptively simple test, which is perfect for the type of system that Lookery uses, provided that team with the data they needed to realize that they had made a good choice in choosing a CDN and that their chosen CDN was able to deliver improved response times when compared to their origin servers.

Check your Cacheability

Cacheability is a nasty word that my spell-checker hates. To define it simply, it refers to the ability of end-user browsers and network-level caching proxies to store and re-use downloaded content based on clear and explicit caching rules delivered in the server response header.

The second Article in David’s series describes how, using Mark Nottingham’s Cacheability Engine, the Lookery team was able to examine the way that the CDNs and the Origin site informed the visitor browser of the cacheability of the JS file that they were downloading.

Cacheability doesn’t seem that important until you remember that most small firms are very conscious of the Bandwidth outlay. These small startups arevery aware when their bandwidth usage reaches 250GB/month level (Lookery’s bandwidth usage at the time the posts were written). Any method that can improve end-user performance while stilll delivering the service they expect is a welcome addition, especially when it is low-cost to free.

In the post, David notes that there appears to be no way in their chosen CDN to modify the Cacheability settings, an issue which appears to have been remedied since the article went up [See current server response headers for the Lookery tag here].

Conclusion

Startups spend a lot of time imagining what success looks like. And when it comes, sometimes they aren’t ready for it, especially when it comes to the ability to handle increasing loads with their often centralized, single-location architectures.

David Cancel, in these two articles, shows how a little early planning, some clear goals, and targeted performance measurement can provide an organization with the information to get them through their initial growth spurt in style.

While this post is aimed at Web performance, the curse of the single metric affects our everyday lives in ways that we have become oblivious to.

When you listen to a business report, the stock market indices are an aggregated metric used to represent the performance of a set group of stocks.

When you read about economic indicators, these values are the aggregated representations of complex populations of data, collected from around the country, or the world.

Sport scores are the final tally of an event, but they may not always represent how well each team performed during the match.

The problem with single metrics lies in their simplicity. When a single metric is created, it usually attempts to factor in all of the possible and relevant data to produce an aggregated value that can represent a whole population of results.

These single metrics are then portrayed as a complete representation of this complex calculation. The presentation of this single metric is usually done in such a way that their compelling simplicity is accepted as the truth, rather than as a representation of a truth.

In the area of Web performance, organizations have fallen prey to this need for the compelling single metric. The need to represent a very complex process in terms that can be quickly absorbed and understand by as large a group of people as possible.

The single metrics most commonly found in the Web performance management field are performance (end-to-end response time of the tested business process) and availability (success rate of the tested business process). These numbers are then merged and transformed by data from a number of sources (external measurements, hit counts, conversions, internal server metrics, packet loss), and this information is bubbled up in an organization. By the time senior management and decision-makers receive the Web performance results, that are likely several steps removed from the raw measurement data.

An executive will tell you that information is a blessing, but only when it speeds, rather than hinders, the decision-making process. A Web performance consultant (such as myself) will tell that basing your decisions on a single metric that has been created out of a complex population of data is madness.

So, where does the middle-ground lie between the data wonks and the senior leaders? The rest of this post is dedicated to introducing a few of the metrics that will, in a small subset of metrics, give a senior leaders better information to work from when deciding what to do next.

A great place to start this process is to examine the percentile distribution of measurement results. Percentiles are known to anyone who has children. After a visit to the pediatrician, someone will likely state that “My son/daughter is in the XXth percentile of his/her age group for height/weight/tantrums/etc”. This means that XX% of the population of children that age, as recorded by pediatricians, report values at or below the same value for this same metric.

Percentiles are great for a population of results like Web performance measurement data. Using only a small set of values, anyone can quickly see how many visitors to a site could be experiencing poor performance.

If at the median (50th percentile), the measured business process is 3.0 seconds, this means that 50% of all of the measurements looked at are being completed in 3.0 seconds or less.

If the executive then looks up to the 90th percentile and sees that it’s at 16.0 seconds, it can be quickly determined that something very bad has happened to affect the response times collected for the 40% of the population between these two points. Immediately, everyone knows that for some reason, an unacceptable number of visitors are likely experiencing degraded and unpredictable performance when they visit the site.

A suggestion for enhancing averages with percentiles is to use the 90th percentile value as a trim ceiling for the average. Then side-by-side comparisons of the untrimmed and trimmed averages can be compared. For sites with a larger number of response time outliers, the average will decrease dramatically when it is trimmed, while sites with more consistent measurement results will find their average response time is similar with and without the trimmed data.

It is also critical to examine the application’s response times and success rates throughout defined business cycles. A single response time or success rate value eliminates

• variations by time of day
• variations by day of week
• variations by month
• variations caused by advertising and marketing

An average is just an average. If at peak buiness hours, response times are 5.0 seconds slower than the average, then the average is meaningless, as business is being lost to poor performance which has been lost in the focus on the single metric.

All of these items have also fallen prey to their own curse of the single metric. All of the items discussed above aggregate the response time of the business process into a single metric. The process of purchasing items online is broken down into discrete steps, and different parts of this process likely take longer than others. And one step beyond the discrete steps are the objects and data that appear to the customer during these steps.

It is critical to isolate the performance for each step of the process to find the bottlenecks to performance. Then the components in those steps that cause the greatest response time or success rate degradations must be identified and targeted for performance improvement initiatives. If there are one or two poorly performing steps in a business process, focusing performance improvement efforts on these is critical, otherwise precious resources are being wasted in trying to fix parts of the application that are working well.

In summary, a single metric provides a sense of false confidence, the sense that the application can be counted on to deliver response times and success rates that are nearly the same as those simple, single metrics.

The average provides a middle ground, a line that says that is the approximate mid-point of the measurement population. There are measurements above and below this average, and you have to plan around the peaks and valleys, not the open plains. It is critical never to fall victim to the attractive charms that come with the curse of the single metric.

The latest “rage” flooding through the social-media world is identi.ca, a Twitter-like micro-blogging service that is built on open-source servers and code.

As with anything that becomes an overnight sensation, the problems of success tend to follow. Using GrabPERF, I have been monitoring the HTML download time of my personal message stream. The results have been interesting.

Much has been made of social-media leaders that says that this is a clone, and that it is slow, etc. But, as has also been noted, it is:

• A one-person operation
• open-source code
• willing to admit that it needs to grow.

So, one-day never makes a performance trend. Over the last week, in my day job, I have watched a large online retailer suffer a similar fate to this newcomer to the social-media arena.

And if everyone who was willing to wait for Twitter to recover waited ten seconds for identi.ca to catch up, then there is a good chance that it may stand a chance of becoming a true competitor, pushing performance improvement.

Plurk was a non-starter for the twitterati. Jaiku has lost momentum, and is failing Google in the same way that Orkut did. And Pownce…what is that?

I hold out high hopes for identi.ca, if only to keep Twitter truly honest.

Yes, it’s YATPBP (Yet Another Twitter Performance Blog Post).

About 10 days ago, I turned GrabPERF loose on Twitter. Now before you accuse this of crippling the service and bringing it to its knees, realize that GrabPERF simply requests a document over HTTP about two times a minute. No additional requests are made for images and the like.

In the ocean of requests coming into the Twitter systems, the GrabPERF requests are like individual water molecules being added to the pool.

The above graph shows performance for the last 24 hours. The purple dots are errors. Complete details, and a dynamically updating graph, can be found here.

Now that I have had a chance to show off, I will leave the Twitter team in peace. I am not a developer or a systems expert. I, like most people, rely on people with specialized skills to analyze and resolve the problem. There are many people on the Web who have taken on the challenge of reverse-engineering Twitter to try and determine how it does what it does, and how they would build a better mousetrap.

Ok. Go do it. Or shut up and let the Twitter team get down to the hard work of making this service work. Or volunteer to help them fix the problems.

The Twitter team has stated that they know how to resolve the issues that are at the heart of the performance issue. But as I said in a comment to @gapingvoid tonight, knowing what the solution is only makes up 15% of the application development process. Building, testing, deploying and verifying the solution takes 85% of the effort.

The Twitter team has a lot of work ahead of them. Buy them beer and pizza and let them get to it.

When FireFox 1.0 was released, the sites hosting this eagerly awaited software effectively were overwhelmed by the number of users attempting to download it.

A colleague of mine referred to it as a Success Failure. What is that? It simply means that you have been so successful at getting the word out and getting people excited about the release that you fail to deliver as a result.

Given the background buzz and increasing frustration of people, it seems that Twitter is having the same thing right now. Question is not so much the cause (broken code? new hardware run amuck? old hardware groaning? a breakpoint in the number of users causing the system to seize up?), but the effect it is having.

High profile users of the service are upset. Regular users are wondering if it’s time to seek an alternative.

Part of the frustration stems from the lack of updates from Twitter itself. For a service that is designed to provide people with flash updates of ongoing events, it appears that they are failing to make use of their own technology and approach. There a semi-secret Twitter Status timeline available (here).

I hope that the team at Twitter get it figured out soon, as I am a new addict to the system. I hate to think that I finally started on this meme at the precise moment that it collapsed.

So, Microsoft is releasing Internet Explorer 8 later this year, and a Beta is available now.

The question is, are you ready for it?

Internally, the tech savvy folks (myself included) have been tossing around the football of how to tackle it. The leap from 2 to 6 connections per host, on top of the basic rendering challenges that come with any new browser release are enough to make a Web team’s collective hive mind melt.

So, the bright team at Gomez are developing an IE 8 readiness kit.

Sounds like a great idea.

Why do you need to be concerned at a network level?

Let’s see…Well, while 6 simultaneous connections may be faster for clients, what happens to the TCP stacks of the devices that handle all of these connections, especially as more and more people start using IE 8?

Imagine that every Firefox user is using FasterFox, and then IE 8 comes along.

Do you have enough TCP sockets?

How quickly are sockets that are ready to be recycled actually recycled?

Are you really ready?

I know that I have been bad at blogging news about GrabPERF, but today there is some. In the last two weeks, we have added two measurement locations: Washington DC AOL and Argentina LaNacion.

Thanks to Carson Evans of AOL, and Jose Falvo and Leonardo Lancellotta of LaNacion for helping out with the installation process.

Want to follow all the Holiday Retail Fun? Check out the GrabPERF Black Friday Dashboard! Follow the Web performance of your favorite retailer all the time!

New and updated for 2008!

This morning, Tiger Direct effectively imploded.

Bad day.

UPDATE: Looks like they found the solution to their problem.

Technorati Tags:
Grabperf, Web performance, Tiger Direct, Cyber Monday

Today’s biggest victim of Black Friday appears to be Sears

Sears measurement data for the last 8 hours can be found here.

UPDATE: It gets worse for Sears.

Technorati Tags:
Grabperf, Sears, Black Friday, Web Performance

Look ma: Macy’s is already too busy.

Yeah. Let the fun begin.

Technorati: GrabPERF “Black Friday” “Web Performance” “Macy’s”