Why GrabPERF?

About four years ago, I had a bright idea that I would like to learn more about how to build and scale a small Web performance measurement platform. I’ve worked in the Web performance industry for nearly a decade now, and this was an experimental platform for me to examine and encounter many of the challenges that I see on a daily basis.

The effort was so successful and garnered enough attention during the initial blogging boom that I was able to sell the whole platform for a tiny (that is not a typo) sum to Technorati.

The name is taken from another experimental tool I wrote called GrabIT2 which uses the PHP cURL libraries to capture timings and HTML data for HTTP requests. It is an extension of my articles and writings on Web performance that started at Webperformance.org, and that have since moved to this blog.

What is GrabERF?

GrabPERF is a multi-location measurement platform, based on PERL, cURL, PHP, and MySQL that is designed to

• Measure the base HTML or a single-object target using HTTP or HTTPS
• Report the data to a central database (located in the San Francisco Area)
• Report the data using a GUI or through text based download

Why not Full Pages with all Objects?

Reason 1: I work for a company that already does that. Lawyers and MBAs among you, do the math.

Reason 2: I am an analyst, not a programmer. The best I can say about my measurement script is hack job.

Why is the GrabPERF interface so clunky?

See reason 2 above.

If you want to write your own interface to the data, let me know.

Why has the interface not changed in nearly three years?

The current interface works. It’s simple, clean, and delivers the data that I and the regular users need to analyze performance issues. If there is something more that you would like to see, let me know!

I like what I see. How can I host a measurement location?

Just contact me, and I can provide you with a list of PERL modules you will need to install on your linux server. In return, I need a static IP address of the machine hosting the measurement agent.

How stable is GrabPERF?

Most of the time, I forget it’s even running. I have logged onto the servers and typed in uptime and discovered that it’s been 6 months or more since the servers have been re-booted.

It was designed to be simple, because that’s all I know how to do. The lack of complexity makes it effectively self-managing.

Shouldn’t all systems be that way?

What if my question isn’t asked / answered here?

Your should know the answer to this by now: contact me.

Tags: experimental tool, GrabIT2, GrabPERF, GrabPERF.org, GUI, HTML, HTTP, Linux, Linux: Server, Perl, PHP, Web Performance, Web performance industry, Web performance measurement platform

Introduction to the Series

This is the first of a four-part series focusing on the reasons why companies measure their Web performance. This perspective is substantially different than ones posited by others in the field as they focus on the meat and potatoes reasons, rather than the sometimes more difficult to imagine future effects that measurement will bring.

Reason One: Customer Generation

It is critical that a company be able to show that their Web services are superior to others, especially in the third-party services and delivery sectors of the Web. In this area, Web performance measurement is key to demonstrating the value and advantage of a service versus the option of self-delivering or using another competitor’s service.

Comparative benchmarking that clearly demonstrates the performance of each of the competitive services in the geographic regions that are of greatest interest to the prospect is key to these Web performance measurements. To achieve truly competitive benchmarks and prove the value of a service, measurements must be realistic and flexible.

In the CDN field, a one object fits all approach is no longer valid. CDNs are responsible for delivering not just images or static objects, but may also host an entire application on their edge servers, serving both HTTP and HTTPS content. In other cases, the application may not be hosted at the edge, but the edge server may act a s a proxy for the application, using advancing routing algorithms to deliver the visitor the requested dynamic content more quickly (in theory) than the origin server alone.

This complex range of services means that a CDN has to be willing to demonstrate effective and efficient service delivery before the sale is complete. A CDN has to be willing to expose their system not just to the backbone-based measurements offered in a traditional customer generation process, but to take measurements from the real-user perspective.

Ad-providers have to be willing to show that their service does not affect the overall performance of the site they are trying to place their content on. Web analytics firms face the same challenge. Web analytics firms have one advantage: if their object doesn’t load properly, it may not effect the visitor experience. However, neither ad-providers nor Web-analytics providers can hide frow Web measurement collection methods that show all of the bling and the blemishes.

Using Web performance measurements to generate customers is a way that a firm can clearly show that they have faith enough in their service to openly compare it to other providers and to the status quo.

But woe be the firm who uses Web performance metrics in a way that tries to show only their good side. Prospects become former prospects very quickly if a firm using Web performance data to generate new business is found to be gaming the system to their advantage. And it will happen.

Customer Generation is a key method that Web performance measurements are used by firms to clearly show how their service is superior to what a prospect currently has, or is also considering. However, this method does come with substantial caveats, including

• The need to measure what is relevant
• The need to measure from where the prospect has the greatest interest
• The need to consider that gaming the system to show advantage will cost a firm in the end.

Tags: firm using Web performance data, frow Web measurement collection methods, HTTP, origin server, Web analytics firms, Web Performance, Web performance measurement, Web performance measurements, Web performance metrics, Web services, Web-analytics providers

The GrabPERF Black Friday Dashboard is done for another year and there were two performance victims that suffered the most at the hands of the onslaught of bargain-hunters in the area of Web performance.

Some caveats that I need to mention about the GrabPERF measurement methodology.

1. Only the base HTML file of each site is measured.
2. Only the base HTML of the homepage is measured. This means that any issues that arose in the shopping process were not captured.

All of the sites in the GrabPERF Holiday Retail Measurement Index can be continually monitored on the GrabPERF Black Friday Dashboard. This page will be available until January 1 2009.

That said, the two primary performance victims this year are HP Shopping and Sears. We focus here on those that did not do that well because sites who have met the Web performance challenge and survived to fight another year are not as interesting from a learning perspective.

HP Shopping

HP Suffered the greatest response time problems, by effectively becoming unresponsive as of 09:00 EST. The greates affect on overall response time came as a result of the First Byte time metric which is a solid proxy for measuring the server or application load, as it is the time between the initial client HTTP request and the server’s HTTP response.

Factored into the poor performance analysis is the fact that GrabPERF only captures data for the base HTML object. If the performance seen here is carried over to the download of all of the graphical content on the page, I would be surprised if anyone was able to make any kind of purchases on the HP web site on Black Friday.

Today, performance has returned to substantially lower levels, indicating that this application was simply not ready for the amount of traffic it received, or ran into a completely unexpected issue when the load increased.

Recommendation for 2009: Load Test the application using this year’s traffic metrics as a baseline for validating the scalability of the application.

Sears

Sears is a returning visitor from last year’s Black Friday measurements. Unfortunately, they return for exactly the same reason that they were on last year - scaling/capacity issues that appear as errors.

And these are the worst kind of errors. As can be seen in the graphic below, the Sears Web site announced to the whole world that they had over-reached and that they could not handle the incoming volume of traffic.

What is interesting is that Sears owns properties that survived the day very well, namely Lands End. The question that must be posed is why does the parent site fail so badly when the child sites handle the traffic without difficulty?

Recommendation for 2009: Load testing for capacity, and meeting with the Lands End team to understand what they are doing to handle the load.

Tags: Black Friday, Black Friday 2008, GrabPERF, HP Shopping, HTML, HTTP, Lands End, Sears, Web Performance, Web performance challenge, Web performance measurement

Once again, the Grabperf Black Friday Dashboard is up and running!

GrabPERF Black Friday Online Retailer Performance Dashboard

Check out your favorite retailer or suggest one we missed!

Tags: Black Friday, Black Friday 2008, GrabPERF, Online Retailer Performance Dashboard, Retailers, Web Performance

I fired up the Boston FIoS measurement location today after a couple of days off, and found that suddenly FIoS doesn’t like the BitTorrent.

The line of purple dots all indicate measurements that reported an error code. All of those measurements come from Boston FiOS. See the real-time graph here.

Accident? Design? That I cannot comment on. I simply report on what I see.

Tags: bittorrent, FIoS, GrabPERF, Verizon, Verizon FIoS, Web Performance, Web performance measurement

In the last 24 hours, thanks to the help of some willing volunteers, GrabPERF has seen the addition of three new measurement locations:

• Dallas, TX (USA)
• Virginia (USA)
• London, UK

All of these location have been graciously provided by the team at e-planning.

Thanks to all of you who volunteer your machines and bandwidth for this project.

As always, we are looking for as more measurement locations. It would be great if we could get some data from the Asia-Pacific region.

Tags: Asia-Pacific, Dallas, e-planning, e-planning.net, External Web Performance, GrabPERF, London, United Kingdom, United States, Virginia, Web Performance, Web performance measurement, West Virginia

Today’s Web interfaces are all about the Flash (literally). Smooth charting, cool effects, callouts to references — ways to try and simplify complex data collections.

Problem-solving and diagnosis requires a far deeper dive than the flashiest interface could ever provide, because it comes down to the numbers. The actual measurements that make up the flashy chart. If you look at the data used by a professional trader and a someone at home looking at stock charts, there is a substantial difference.

When you get down to that level of analysis, the interface becomes irrelevant. Any analyst worth her or his salary (or salt - same thing) can tell you more from a spreadsheet full of relevant numbers than they can from any pretty graphic. This is true in any field.

When do traders or Web performance analysts use pretty charts? When they have to explain complex issues to non-technical or non-specialist audiences. When these analysts work on solving the sticky problems faced in the everyday world, they always fall back on the numbers.

Web performance data consists of the same few components, regardless of which company is providing the data. In effect, beyond a few key pieces of information about how the measurement data is captured, all Web performance data is the same.

Just because the components that make up the data are the same does not guarantee that the data from two different providers is of the same quality. In an imaginary system, Web performance data from all the major providers could flow into a centralized repository and be transformed using an XSLT or some other mangler so that it would be indistinguishable in most cases to tell which firm was the source.

But a skilled analyst would quickly learn to recognize the data that can be trusted. That would be the data that quickly and accurately represented the issues he was trying to diagnose. The data that flowed with the known patterns of the Web site. The data that helped him do his job more effectively.

In the end, a pretty interface can go a long way to hide the quality of the data that is being represented. A shiny gloss on poor data does not make it better data. It is critical that the data that underlies that pretty chart is able to live up to the quality demands of the people who use it every day.

Selling the interface is selling the brand. Trust in the data builds the reputation.

Which one sold you when you chose your Web performance measurement provider?

Tags: branding, data, flash, interface, measurements, reputation, Web interfaces, Web Performance, Web performance analysts, web performance data, Web performance measurement, Web performance measurement data, XSLT

Yesterday in the Fast Company Live Fail Whale session [mention on Scoble's blog here], Paul Bucheit of FriendFeed jokingly said that his company’s external alerting mechanism was Louis Gray.

I cringed when I read that, as the last people who should be letting you know you have an issue are your visitors or customers. I know that FriendFeed is new and may not have the ops team that Dorion Carroll and Technorati have developed over the years, but it is still critical.

You have done a lot as a company to build a brand. Don’t let your internal and external performance sully your reputation. There are a number of low-cost and free ways to watch your performance and alert you before things break.

Louis Gray is a great guy. But he is not an objective and reliable way to alert you when something is wrong with your site.

Tags: alerting, Dorion Carroll, External Web Performance, friendfeed, Internal Web Performance, Louis Gray, Paul Bucheit, Technorati, Web Performance

This is the year I turn 40. As a result, I am looking back upon my life, my career, and trying to determine what I do best. If I could make my life into an elevator pitch, what would it be?

I decided to take what I do right now and see how low I could take it. What does my career boil down to?

It came down to three simple words: Educate, Guide, and Solve.

Each of these describes a facet of my career that provides a profound sense of personal satisfaction. Each of these is unique in that they give me the chance to share what I know with others, while still gaining new experiences in the process.

These three things are simultaneously selfish and selfless. I believe that in order to have a successful, productive, and fulfilling career, these three things need to serve as the foundation of everything I do.

Educate

I work in a small community of Web performance analysts. I have spent years training myself to see the world through the eyes of a Web site and how it presents to the outside world. As I taught myself to see the world this way, I was asked to share what I knew with others.

At first I did this through technical support and a training course I helped develop. Then I moved into consulting. I began to blog and comment on Web performance.

I needed to share what I knew with others, because it is meaningless to hoard all of your knowledge. While I am paid well as a consultant, it is also important that as many people as possible learn from me; and that doesn’t always need to sold to the highest bidder.

Guide

While some may say that there is no difference between Guide and Educate, I see a profound chasm between the two.

We have all been educated at some point. We have sat through classes and lectures and labs that convey information to us, and have provided the foundation for what we know.

But we have also encountered people who have shown us how to step beyond the information. They place the information that they are giving us in a larger context, allow us to see problems as a component of the whole.

That is what I strive to do. Not only do I want to give people the functional tools they need to interpret the data, I want them to then take that data and see the patterns in the data. I work closely with colleagues and customers, helping them see the patterns, understand how they tie to the things I say everyday, and then be able to solve this type of problem on their own the next time.

A guide is only useful when the path is not known. Once I have showed someone the path, I can return to my place, in the knowledge that they are as experienced on the path as I am.

Solve

Once you have shown someone what the data can do, how to see the patterns, it is critical that they have an understand how to take that pattern and change it for the better. Seeing a pattern and understanding its cause are only the beginning.

I can share my experiences, share how others have solved problems similar to this one, help them fix the problem.

And then be able to show that the problem is solved. An unmeasured, yet resolved problem, is meaningless.

Summary

This is the skeletal description of what I want to achieve in my career. I could expand these topics for a lot longer, but the question I propose is: What three concepts can you boil your career down to?

Tags: career, educate, functional tools, guide, solve, three concepts, three principles, three things, Web Performance, Web performance analysts, Work

The Dog and The Toolbox

One day, a dog stumbled upon a toolbox left on the floor. There was a note on it, left by his master, which he couldn’t read. He was only a dog, after all.

He sniffed it. It wasn’t food. It wasn’t a new chew toy. So, being a good dog, he walked off and lay on his mat, and had a nap.

When the master returned home that night, the dog was happy and excited to see him. He greeted his master with joy, and brought along his favorite toy to play with.

He was greeted with yelling and anger and “bad dog”. He was confused. What had he done to displease his master? Why did the master keep yelling at him, and pointing at the toolbox. He had been good and left it alone. He knew that it wasn’t his.

With his limited understanding of human language, he heard the words “fix”, “dishwasher”, and “bad dog”. He knew that the dishwasher was the yummy cupboard that all of the dinner plates went in to, and came out less yummy and smelling funny.

He also knew that the cupboard had made a very loud sound that had scared the dog two nights ago, and then had spilled yucky water on the floor. He had barked to wake his master, who came down, yelling at the dog, then yelling at the machine.

But what did fix mean? And why was the master pointing at the toolbox?

The Toolbox and Web Performance

It is far too often that I encounter companies that have purchased Web performance service that they believe will fix their problems. They then pass the day-to-day management of this information on to a team that is already overwhelmed with data.

What is this team supposed to do with this data? What does it mean? Who is going to use it? Does it make my life easier?

When it comes time to renew the Web performance services, the company feels gipped. And they end up yelling at the service company who sold them this useless thing, or their own internal staff for not using this tool.

To an overwhelmed IT team, Web performance tools are another toolbox on the floor. They know it’s there. It’s interesting. It might be useful. But it makes no sense to them, and is not part of what they do.

Giving your dog the toolbox does not fix your dishwasher. Giving an IT team yet another tool does not improve the performance of a Web site.

Only in the hands of a skilled and trained team does the Web performance of a site improve, or the dishwasher get fixed. As I have said before, a tool is just a tool. The question that all organizations must face is what they want from their Web performance services.

Has your organization set a Web performance goal? How do you plan to achieve your goals? How will you measure success? Does everyone understand what the goal is?

After you know the answers to those questions, you will know that that as amazing as he is, your dog will not ever be able to fix your dishwasher.

But now you know who can.

Tags: Web Performance, Web performance goal, Web performance service, Web performance services, Web performance tools