Why GrabPERF?

About four years ago, I had a bright idea that I would like to learn more about how to build and scale a small Web performance measurement platform. I’ve worked in the Web performance industry for nearly a decade now, and this was an experimental platform for me to examine and encounter many of the challenges that I see on a daily basis.

The effort was so successful and garnered enough attention during the initial blogging boom that I was able to sell the whole platform for a tiny (that is not a typo) sum to Technorati.

The name is taken from another experimental tool I wrote called GrabIT2 which uses the PHP cURL libraries to capture timings and HTML data for HTTP requests. It is an extension of my articles and writings on Web performance that started at Webperformance.org, and that have since moved to this blog.

What is GrabERF?

GrabPERF is a multi-location measurement platform, based on PERL, cURL, PHP, and MySQL that is designed to

• Measure the base HTML or a single-object target using HTTP or HTTPS
• Report the data to a central database (located in the San Francisco Area)
• Report the data using a GUI or through text based download

Why not Full Pages with all Objects?

Reason 1: I work for a company that already does that. Lawyers and MBAs among you, do the math.

Reason 2: I am an analyst, not a programmer. The best I can say about my measurement script is hack job.

Why is the GrabPERF interface so clunky?

See reason 2 above.

If you want to write your own interface to the data, let me know.

Why has the interface not changed in nearly three years?

The current interface works. It’s simple, clean, and delivers the data that I and the regular users need to analyze performance issues. If there is something more that you would like to see, let me know!

I like what I see. How can I host a measurement location?

Just contact me, and I can provide you with a list of PERL modules you will need to install on your linux server. In return, I need a static IP address of the machine hosting the measurement agent.

How stable is GrabPERF?

Most of the time, I forget it’s even running. I have logged onto the servers and typed in uptime and discovered that it’s been 6 months or more since the servers have been re-booted.

It was designed to be simple, because that’s all I know how to do. The lack of complexity makes it effectively self-managing.

Shouldn’t all systems be that way?

What if my question isn’t asked / answered here?

Your should know the answer to this by now: contact me.

Tags: experimental tool, GrabIT2, GrabPERF, GrabPERF.org, GUI, HTML, HTTP, Linux, Linux: Server, Perl, PHP, Web Performance, Web performance industry, Web performance measurement platform

In 2004-2005, as a lark, I created my own Web performance measurement system, using PERL, PHP and MySQL. In August 2005, I managed to figure out how to include remote agents.

I dubbed it…GrabPERF. An odd name, but an amalgamation of “Grab” and “Performance” that made sense to my mind at the time. I also never though that it would go beyond my house, a couple of basement servers, and a cable modem.

In the intervening three years, I have managed to:

• scale the system to handle over 250 individual measurements
• involve nine remote measurement locations
• move the system to the Technorati datacenter
• provide key operational measurement data to system visitors

Although the system lives in the Technorati datacenter and is owned by them, I provide the majority of the day-to-day maintenance on a volunteer basis, if only to try and keep my limited coding skills up.

But this post is not about me. It’s about GrabPERF.

Thanks to the help of a number of volunteers, I have measurement locations in the San Francisco Bay Area, Washington DC, Boston, Portugal, Germany and Argentina.

While this is a good spread, I am still looking to gather volunteers who can host a GrabPERF measurement location. The areas where GrabPERF has the most need are:

• Asia-Pacific
• South Asia (India, Pakistan, Bangladesh)
• UK and Continental Europe
• Central Europe, including the ancestral homeland of Polska

It would also be great to get a funky logo for the system, so if you are a graphic designer and want to create a cool GrabPERF logo, let me know.

The current measurement system requires Linux, cURL and a few add-on Perl modules. I am sure that I could work on other operating systems, I just haven’t had the opportunity to experiment.

If you or your organization can help, please contact me using the GrabPERF contact form.

Tags: Argentina, Asia-Pacific, Bangladesh, Boston, cable modem, Central Europe, Germany, GrabPERF, GrabPERF.org, India, internet performance measurement data, Linux, mysql, operating systems, Pakistan, performance measurement, Perl, PHP, Polska, Portugal, San Francisco, United Kingdom, Washington DC, web, Web Performance, Web performance measurement data, Web performance measurement system

Recently there was an outage at a hit-tracking vendor I was using to track the hits on my externally hosted blog, leaving me with a gap in my visitor data several hours long. While this was an inconvenience for me, I realized that this could be mission critical failure to an online business reliant on this data.

To resolve this, I used the PHP HTTP environment variables and the built-in function for converting IP addresses to IP numbers to create my own hit-tracker. It is a rudimentary tracking tool, but it provides me with the basic information I need to track visitors.

To begin, I wrote a simple PHP script to insert tracking data into a MySQL database. How do you do that? You use the gd features in PHP to draw an image, and insert the data into the database.

---

header ("Content-type: image/png");

include("dbconnect_logger.php");
$logtime = date("YmdHis");
$ipquery = sprintf("%u",ip2long($_SERVER['REMOTE_ADDR']));

        $query2 = "INSERT into logger.blog_log values \
               ($logtime,$ipquery,'$HTTP_USER_AGENT','$HTTP_REFERER')";
        mysql_query($query2) or die("Log Insert Failed");

mysql_close($link);

$im = @ImageCreate (1, 1)
or die ("Cannot Initialize new GD image stream");
$background_color = ImageColorAllocate ($im, 224, 234, 234);
$text_color = ImageColorAllocate ($im, 233, 14, 91);

// imageline ($im,$x1,$y1,$x2,$y2,$text_color);
imageline ($im,0,0,1,2,$text_color);
imageline ($im,1,0,0,2,$text_color);

ImagePng ($im);
?>

---

Next, I created the database table.

---

DROP TABLE IF EXISTS `blog_log`;
CREATE TABLE `blog_log` (
  `date` timestamp NOT NULL default '0000-00-00 00:00:00',
  `ip_num` double NOT NULL default '0',
  `uagent` varchar(200) default NULL,
  `visited_page` varchar(200) NOT NULL default '',
  UNIQUE KEY `date` (`date`,`ip_num`,`visited_page`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

---

It’s done. I can now log any request I want using this embedded tracker.

Data should begin flowing to your database immediately. This sample snippet of code will allow you to pull data for a selected day and list each individual hit.

---

$query1 = "SELECT
                bl.ip_num,
                DATE_FORMAT(bl.date,'%d/%b/%Y %H:%i:%s') AS NEW_DATE,
                bl.uagent,
                bl.visited_page
        FROM blog_log bl
        WHERE
                DATE_FORMAT(bl.date,'%Y%m%d') ='$YMD'
		and uagent not REGEXP '(.*bot.*|.*crawl.*|.*spider.*|^-$|.*slurp.*|.*walker.*|.*lwp.*|.*teoma.*|.*aggregator.*|.*reader.*|.*libwww.*)'
        ORDER BY bl.date ASC";

print "<table border=\"1\">\n";
print "<tr><td>IP</td><td>DATE</td><td>USER-AGENT</td><td>PAGE VIEWED</td></tr>";
while ($row = mysql_fetch_array($result1)) {
        $visitor = long2ip($row[ip_num]);
        print "<tr><td>$visitor</td><td nowrap>$row[NEW_DATE]</td><td nowrap>$row[uagent]</td><td>";

	if ($row[visited_page] == ""){
    	    print " --- </td></tr>\n";
	} else {
    	    print "<a href=\"$row[visited_page]\" target=\_blank\">$row[visited_page]</a></td></tr>\n";
	}

}

mysql_close($link);

---

And that’s it. A few lines of code and you’re done. With a little tweaking, you can integrate the IP number data with a number of Geographic IP databases available for purchase to track by country and ISP, and using graphics applications for PHP, you can add graphs.

For my own purposes, this is an extension of the Geographic IP database I created a number of years ago. This application extracts IP address information from the five IP registrars, and inserts it into a database. Using the log data collected by the tracking bug above and the lookup capabilities of the Geographic IP database, I can quickly track which countries and ISP drive the most visitors to my site, and use this for general interest purposes, as well as the ability to isolate any malicious visitors to the site.

Tags: analytics, database, graphics applications, hits, HTTP, IP, ISP, online, Perl, PHP, Registrar, rudimentary tracking tool, tracking, traffic, visitors

 

 

Don’t really need to explain this, do we?

Via Core77 and LuxuryLaunches

Tags: ASP, blog, content, HTTP, IM, ISP, it, Om, PHP, San Francisco, Technorati

Today’s biggest victim of Black Friday appears to be Sears

Sears measurement data for the last 8 hours can be found here.

UPDATE: It gets worse for Sears.

Technorati Tags:
Grabperf, Sears, Black Friday, Web Performance

Tags: app, data, GrabPERF, GrabPERF.org, HTTP, IE, IM, it, LAN, measurement, Om, performance, PHP, Technorati, update, web, Web Performance, Web+performance

mon.itor.us, a service which also provides free Web performance measurement services, appears to be having a wee problem.

The most recent GrabPERF data on this site is available here. The issue may be corrected by the time you look at the data.

I don’t wish suffering like this on anyone. GrabPERF had it’s own 3-4 day outage a few months ago. It’s just sad to see when monitoring services go down.

Tags: app, ARIN, data, GrabPERF, GrabPERF.org, HTTP, IE, IM, it, measurement, Om, One, outage, performance, performance measurement, PHP, pr, web, Web Performance, Web performance measurement

Last night, I got motivated.

Ok, I got manic. Goes with my life.

As a part of that mania, I had a breakthrough on how to present GrabPERF data that I’ve actually been collecting for nearly a year: text match failures.

GrabPERF has the ability to match text on page results using a standard PERL regex. By putting a regex into the measurement configuration, I can confirm that the data downloaded matches what should be there.

If there is not a match, the headers and page text are captured and inserted into a table. Up until yesterday, I was the only one who could view the data. Now, if you go to the graph configuration page (http://grabperf.org/measure_page.php?test=[insert test id here]), and see the following type of result, then click through the links.

If your graph configuration page says no text match configured, and you want one, let me know!

Tags: ARIN, content, data, EAD, GrabPERF, GrabPERF.org, HTTP, IE, IM, it, measurement, Om, One, Perl, PHP, pr

If you use GrabPERF on a regular basis, the somewhat flaky navigation method has become second nature to you. In fact, to circumvent some of the idiosyncrasies, you have probably bookmarked your favourite pages.

Yesterday, I broke your links.

When I redesigned GrabPERF in February 2006, I had just discover the require function in PHP, and decided to build the entire the structure using a single container page as the framework, and individual functions called using URL parameters.

As time went on, my own “brilliance” started to get in the way of maintaining and updating the code. It took me 10-15 minutes to figure out how I constructed pages, and then find the right code to fix or update.

Yesterday, I got completely fed up with this structure.

Now, all functions have their own unique pages, making maintenance a snap. And as an added benefit, I can now effectively track the usage of individual pages, so I know where to through development efforts.

Some of the changes.

http://grabperf.org/homepage.php?page=compare&test=2&tests%5B%5D=276&tests%5B%5D=277&tests%5B%5D=279&tests%5B%5D=280

becomes

http://grabperf.org/compare.php?test=2&tests%5B%5D=276&tests%5B%5D=277&tests%5B%5D=279&tests%5B%5D=280

---

http://grabperf.org/homepage.php?page=scatter&test=277&hours=2

becomes

http://grabperf.org/scatter.php?test=277&hours=2

 

I apologize for the confusion that this may cause, but in the long run, this will help me make the code better, and more robust.

Tags: book, content, design, GrabPERF, GrabPERF.org, HTTP, IE, IM, improvement, ISP, it, maintain, maintenance, Om, optimize, Pages, performance, PHP, pr, run, Technorati, update, web, Web Performance

As many readers know, I am going through the process — if you call filing a bunch of paperwork and not hearing anything for 2 years a process — of obtaining Permanent Residency in the United States, often referred to as the Green Card.

This morning, on NPR, there was a story about a foul-up in the processing of Green Cards that is suspicious, to say the least.

I have started referring to this process as the Dream Card because it leaves one thinking that the application they completed was done in a dream, a long time ago. An like most dreams, it is a fable of the subconscious mind and as likely to come true as those blue, flying penguins in my dream last night.

The degree of complexity that accompanies the application process has made bureaucrats from the Byzantine Empire write letters of complaint to their members of Congress, saying that the USCIS is giving them a bad name. Kafka has been seen rising from the dead at night, and penning a new tale based on this experience.

Other people covering this story.

NY Times
The Guardian
Times Of India
Miami Herald
San Jose Mercury News
Sacramento Bee Editorial

A few media outlets have grabbed this story as an example of just how broken the US system is when it comes to immigration, especially given the irony of the recent debate over the immigration bill that was tossed out of Congress. How could the immigration system have hoped to deal with the new regulations, if thousands, perhaps tens of thousands, of valid visas go unused every year, due to government inefficiency.

Why would an illegal immigrant bother to go through a legal process that punishes the very people who are taking the time to follow the rules?

I would raise my voice in protest; but it would do no good. Drawing a pool of highly skilled, well compensated indentured servants from around the world to these shores to keep the wheels of innovation and development rolling appears to have become the American way.

And like indentured servants everywhere, we are a disposable commodity, to be teased by the promise that some day, we could, we might, just maybe be able to live here (and still not be able to vote) as Permanent Residents.

Tags: Green Card, fiasco, scandal, State Department, USCIS, U.S. Citizenship and Immigration Services, indentured, servants, work visa, visa

Tags: app, application, ARIN, dreams, EAD, government, Green Card, HTML, HTTP, IE, IM, Immigration, India, IP, ISP, it, leave, live, media, NY Times, Om, One, opinion, permanent resident, PHP, pr, Technorati, United States

I have set up measurements to monitor the main pages of some of the world’s largest mobile phone providers.

• US Providers
• Canadian Providers
• European Providers
• AsiaPac / South Asia Providers

Just something to do on a rainy Sunday.

Tags: GrabPERF, Web performance, GSM, CDMA, Web site, mobile, wireless, US, Asia, Europe

Tags: Asia, Europe, GrabPERF, GrabPERF.org, GSM, HTTP, it, measurement, metrics, Om, One, Pages, performance, PHP, pr, Technorati, web, Web Performance, Web+performance