The line of purple dots all indicate measurements that reported an error code. All of those measurements come from Boston FiOS. See the real-time graph here.

Accident? Design? That I cannot comment on. I simply report on what I see.

The data from this period has been cut and hourly averages have been re-calculated.

We apologize for the inconvenience.

I dubbed it…GrabPERF. An odd name, but an amalgamation of “Grab” and “Performance” that made sense to my mind at the time. I also never though that it would go beyond my house, a couple of basement servers, and a cable modem.

In the intervening three years, I have managed to:

• scale the system to handle over 250 individual measurements
• involve nine remote measurement locations
• move the system to the Technorati datacenter
• provide key operational measurement data to system visitors

Although the system lives in the Technorati datacenter and is owned by them, I provide the majority of the day-to-day maintenance on a volunteer basis, if only to try and keep my limited coding skills up.

But this post is not about me. It’s about GrabPERF.

Thanks to the help of a number of volunteers, I have measurement locations in the San Francisco Bay Area, Washington DC, Boston, Portugal, Germany and Argentina.

While this is a good spread, I am still looking to gather volunteers who can host a GrabPERF measurement location. The areas where GrabPERF has the most need are:

• Asia-Pacific
• South Asia (India, Pakistan, Bangladesh)
• UK and Continental Europe
• Central Europe, including the ancestral homeland of Polska

It would also be great to get a funky logo for the system, so if you are a graphic designer and want to create a cool GrabPERF logo, let me know.

The current measurement system requires Linux, cURL and a few add-on Perl modules. I am sure that I could work on other operating systems, I just haven’t had the opportunity to experiment.

If you or your organization can help, please contact me using the GrabPERF contact form.

Over the history of ther service, we have been steadily increasing the number of measurements we take each day.

The large leap occurred when a very large number of tests were added to the system on a single day. But based on this data, the system is gathering more than 900,000 measurements every day.

Thanks to all of the people who volunteer their machines and bandwidths to support this effort!

TechCrunch has a small issue.

It’s likely they’ll be back soon, but it’s still an interesting thing to see.

Update - 09:17 EDT (13:17 GMT)

TechCrunch is back up as of 08:49 EDT (12:49 GMT).

Many thanks to the Technorati team for working hard to resolve this complex issue, which eventually turned out to be a loose fiber-cable.

Data from the time period has been cut, so no one’s stats should be affected.

I apologize for the incovenience.

About 10 days ago, I turned GrabPERF loose on Twitter. Now before you accuse this of crippling the service and bringing it to its knees, realize that GrabPERF simply requests a document over HTTP about two times a minute. No additional requests are made for images and the like.

In the ocean of requests coming into the Twitter systems, the GrabPERF requests are like individual water molecules being added to the pool.

The above graph shows performance for the last 24 hours. The purple dots are errors. Complete details, and a dynamically updating graph, can be found here.

Now that I have had a chance to show off, I will leave the Twitter team in peace. I am not a developer or a systems expert. I, like most people, rely on people with specialized skills to analyze and resolve the problem. There are many people on the Web who have taken on the challenge of reverse-engineering Twitter to try and determine how it does what it does, and how they would build a better mousetrap.

Ok. Go do it. Or shut up and let the Twitter team get down to the hard work of making this service work. Or volunteer to help them fix the problems.

The Twitter team has stated that they know how to resolve the issues that are at the heart of the performance issue. But as I said in a comment to @gapingvoid tonight, knowing what the solution is only makes up 15% of the application development process. Building, testing, deploying and verifying the solution takes 85% of the effort.

The Twitter team has a lot of work ahead of them. Buy them beer and pizza and let them get to it.

Thanks to Carson Evans of AOL, and Jose Falvo and Leonardo Lancellotta of LaNacion for helping out with the installation process.

Sears measurement data for the last 8 hours can be found here.

UPDATE: It gets worse for Sears.

Technorati Tags:
Grabperf, Sears, Black Friday, Web Performance

Yeah. Let the fun begin.

Technorati: GrabPERF “Black Friday” “Web Performance” “Macy’s”