A comment on my Hit Tracking with PHP and MySQL post raised some interesting questions about what a consumer of Web analytics data needs to consider when selecting providers for their sites.

Most folks are familiar with the model of Web analytics vendors: You place their JS tag on your page which makes a call back to their centralized system with all of the data that can be collected about the visitor who has just come to your site.

There are three items that you, the consumer of this third-party service, need to get straight answers about.

Performance of the Tag

Visitors to your site do not know which components you are responsible for and which you have farmed out to vendors and services. In their mind, all content is your content. This means that the performance of the Web analytics tag, which if not placed correctly on the page can affect the rendering of the content, is a critical factor in selecting a vendor.

To my knowledge, none of the Web analytics vendors used by most people blog and social media firms (StatCounter, GetClicky, Google Analytics, Omniture, etc.) freely share response time and error rate metrics of their tag infrastructure with the world as measured by a third party.

Is the tag delivered from a central location, or does the provider use a CDN? Is the data delivered asynchronously or synchronously?

The download performance of the tag is not the only concern. Ask them to provide data on the average processing time that they see when a client parses and processes the code. How does this processing time vary from browser to browser, between OSs? What steps are they taking to ensure that the tag does not affect the perceived performance of your page?

Ask for this data. Ask that the measurement data be collected by an impartial third party. Demand that this data be freely available to you before you make a decision on purchasing their service.

Size of the Tag

All of these services rely on a JS tag to collect and deliver the data to their data warehouse. Ask the provider to tell you how large this tag is, and what steps they have taken to reduce its size so that it has a limited effect on the download of your content.

Have they minified the tag? Is it delivered using HTTP compression to further reduce the size? Are ways to reduce the size of the tag always under considerations?

Data Storage

Analyzing data takes a couple of forms: daily operational views to spot changes or issues; and long-term trending to find larger patterns and major shifts in your visitors. As a result, data needs to be available with a substantial degree of detail while still providing aggregated data that allows larger patterns to be easily discovered.

How long does your provider store detailed data? What is the data expiry policy? Can you extract the data and import it into your own database?

Summary

Web analytics is key to determining what works and doesn’t work on your site. It tells who, where and how people are accessing your content. But without a Web analytics partner/vendor that provides performance and support, you may be left more in the cold than if you were just looking at your raw Web logs.

For a number of years, I have owned three very popular domain names: WebPerformance.org, WebCaching.org, and WebCompression.org. Last night, after many days of consideration, I stopped pointing them at their own distinct Web space and pointed them at this blog.

This is not a bad or evil thing, considering that for at least 18 months, the articles that were hosted at those sites were duplicated here in a more manageable format.

For those who have come looking for the content from those sites, it is here. The search box in the right column can help you locate it.

But for those who would like a refresher, here is a list of the most popular articles on this blog, as selected by Web performance traffic.

Web Performance Concepts Series

• Web Performance, Part I: Fundamentals
• Web Performance, Part II: What are you calling ‘average’?
• Web Performance, Part III: Moving Beyond Average
• Web Performance, Part IV: Finding The Frequency
• Web Performance, Part V: Baseline Your Data
• Web Performance, Part VI: Benchmarking Your Site
• Web Performance, Part VII: Reliability and Consistency
• Web Performance, Part VIII: How do you define fast?
• Web Performance, Part IX: Curse of the Single Metric

Why Web Measurement Series

• Why Web Measurement, Part I: Customer Generation
• Why Web Measurement, Part II: Customer Retention
• Why Web Measurement, Part III: Business Operations
• Why Web Measurement, Part IV: Technical Operations

Web Performance Tuning

• The Dichotomy of the Web: Andy King’s Website Optimization
• Performance Improvement From Compression
• Baseline Testing With cURL
• Compressing Web Output Using mod_deflate and Apache 2.0.x
• Compressing PHP Output
• Using Client-Side Cache Solutions And Server-Side Caching Configurations To Improve Internet Performance
• Performance Improvement From Caching and Compression
• Compressing Web Output Using mod_gzip for Apache 1.3.x and 2.0.x
• mod_gzip Compile Instructions
• Hacking mod_deflate for Apache 2.0.44 and lower

IP Tracking

• Geographic IP database using PERL, PHP and MySQL — UPDATE: September 16 2008
• Hit Tracking with PHP and MySQL

Why GrabPERF?

About four years ago, I had a bright idea that I would like to learn more about how to build and scale a small Web performance measurement platform. I’ve worked in the Web performance industry for nearly a decade now, and this was an experimental platform for me to examine and encounter many of the challenges that I see on a daily basis.

The effort was so successful and garnered enough attention during the initial blogging boom that I was able to sell the whole platform for a tiny (that is not a typo) sum to Technorati.

The name is taken from another experimental tool I wrote called GrabIT2 which uses the PHP cURL libraries to capture timings and HTML data for HTTP requests. It is an extension of my articles and writings on Web performance that started at Webperformance.org, and that have since moved to this blog.

What is GrabERF?

GrabPERF is a multi-location measurement platform, based on PERL, cURL, PHP, and MySQL that is designed to

• Measure the base HTML or a single-object target using HTTP or HTTPS
• Report the data to a central database (located in the San Francisco Area)
• Report the data using a GUI or through text based download

Why not Full Pages with all Objects?

Reason 1: I work for a company that already does that. Lawyers and MBAs among you, do the math.

Reason 2: I am an analyst, not a programmer. The best I can say about my measurement script is hack job.

Why is the GrabPERF interface so clunky?

See reason 2 above.

If you want to write your own interface to the data, let me know.

Why has the interface not changed in nearly three years?

The current interface works. It’s simple, clean, and delivers the data that I and the regular users need to analyze performance issues. If there is something more that you would like to see, let me know!

I like what I see. How can I host a measurement location?

Just contact me, and I can provide you with a list of PERL modules you will need to install on your linux server. In return, I need a static IP address of the machine hosting the measurement agent.

How stable is GrabPERF?

Most of the time, I forget it’s even running. I have logged onto the servers and typed in uptime and discovered that it’s been 6 months or more since the servers have been re-booted.

It was designed to be simple, because that’s all I know how to do. The lack of complexity makes it effectively self-managing.

Shouldn’t all systems be that way?

What if my question isn’t asked / answered here?

Your should know the answer to this by now: contact me.

In 2004-2005, as a lark, I created my own Web performance measurement system, using PERL, PHP and MySQL. In August 2005, I managed to figure out how to include remote agents.

I dubbed it…GrabPERF. An odd name, but an amalgamation of “Grab” and “Performance” that made sense to my mind at the time. I also never though that it would go beyond my house, a couple of basement servers, and a cable modem.

In the intervening three years, I have managed to:

• scale the system to handle over 250 individual measurements
• involve nine remote measurement locations
• move the system to the Technorati datacenter
• provide key operational measurement data to system visitors

Although the system lives in the Technorati datacenter and is owned by them, I provide the majority of the day-to-day maintenance on a volunteer basis, if only to try and keep my limited coding skills up.

But this post is not about me. It’s about GrabPERF.

Thanks to the help of a number of volunteers, I have measurement locations in the San Francisco Bay Area, Washington DC, Boston, Portugal, Germany and Argentina.

While this is a good spread, I am still looking to gather volunteers who can host a GrabPERF measurement location. The areas where GrabPERF has the most need are:

• Asia-Pacific
• South Asia (India, Pakistan, Bangladesh)
• UK and Continental Europe
• Central Europe, including the ancestral homeland of Polska

It would also be great to get a funky logo for the system, so if you are a graphic designer and want to create a cool GrabPERF logo, let me know.

The current measurement system requires Linux, cURL and a few add-on Perl modules. I am sure that I could work on other operating systems, I just haven’t had the opportunity to experiment.

If you or your organization can help, please contact me using the GrabPERF contact form.

Recently there was an outage at a hit-tracking vendor I was using to track the hits on my externally hosted blog, leaving me with a gap in my visitor data several hours long. While this was an inconvenience for me, I realized that this could be mission critical failure to an online business reliant on this data.

To resolve this, I used the PHP HTTP environment variables and the built-in function for converting IP addresses to IP numbers to create my own hit-tracker. It is a rudimentary tracking tool, but it provides me with the basic information I need to track visitors.

To begin, I wrote a simple PHP script to insert tracking data into a MySQL database. How do you do that? You use the gd features in PHP to draw an image, and insert the data into the database.

header ("Content-type: image/png");

include("dbconnect_logger.php");
$logtime = date("YmdHis");
$ipquery = sprintf("%u",ip2long($_SERVER['REMOTE_ADDR']));

        $query2 = "INSERT into logger.blog_log values \
               ($logtime,$ipquery,'$HTTP_USER_AGENT','$HTTP_REFERER')";
        mysql_query($query2) or die("Log Insert Failed");

mysql_close($link);

$im = @ImageCreate (1, 1)
or die ("Cannot Initialize new GD image stream");
$background_color = ImageColorAllocate ($im, 224, 234, 234);
$text_color = ImageColorAllocate ($im, 233, 14, 91);

// imageline ($im,$x1,$y1,$x2,$y2,$text_color);
imageline ($im,0,0,1,2,$text_color);
imageline ($im,1,0,0,2,$text_color);

ImagePng ($im);
?>

DROP TABLE IF EXISTS `blog_log`;
CREATE TABLE `blog_log` (
  `date` timestamp NOT NULL default '0000-00-00 00:00:00',
  `ip_num` double NOT NULL default '0',
  `uagent` varchar(200) default NULL,
  `visited_page` varchar(200) NOT NULL default '',
  UNIQUE KEY `date` (`date`,`ip_num`,`visited_page`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Data should begin flowing to your database immediately. This sample snippet of code will allow you to pull data for a selected day and list each individual hit.

$query1 = "SELECT
                bl.ip_num,
                DATE_FORMAT(bl.date,'%d/%b/%Y %H:%i:%s') AS NEW_DATE,
                bl.uagent,
                bl.visited_page
        FROM blog_log bl
        WHERE
                DATE_FORMAT(bl.date,'%Y%m%d') ='$YMD'
		and uagent not REGEXP '(.*bot.*|.*crawl.*|.*spider.*|^-$|.*slurp.*|.*walker.*|.*lwp.*|.*teoma.*|.*aggregator.*|.*reader.*|.*libwww.*)'
        ORDER BY bl.date ASC";

print "<table border=\"1\">\n";
print "<tr><td>IP</td><td>DATE</td><td>USER-AGENT</td><td>PAGE VIEWED</td></tr>";
while ($row = mysql_fetch_array($result1)) {
        $visitor = long2ip($row[ip_num]);
        print "<tr><td>$visitor</td><td nowrap>$row[NEW_DATE]</td><td nowrap>$row[uagent]</td><td>";

	if ($row[visited_page] == ""){
    	    print " --- </td></tr>\n";
	} else {
    	    print "<a href=\"$row[visited_page]\" target=\_blank\">$row[visited_page]</a></td></tr>\n";
	}

}

mysql_close($link);

For my own purposes, this is an extension of the Geographic IP database I created a number of years ago. This application extracts IP address information from the five IP registrars, and inserts it into a database. Using the log data collected by the tracking bug above and the lookup capabilities of the Geographic IP database, I can quickly track which countries and ISP drive the most visitors to my site, and use this for general interest purposes, as well as the ability to isolate any malicious visitors to the site.

Don’t really need to explain this, do we?

Via Core77 and LuxuryLaunches

Today’s biggest victim of Black Friday appears to be Sears

Sears measurement data for the last 8 hours can be found here.

UPDATE: It gets worse for Sears.

Technorati Tags:
Grabperf, Sears, Black Friday, Web Performance

mon.itor.us, a service which also provides free Web performance measurement services, appears to be having a wee problem.

The most recent GrabPERF data on this site is available here. The issue may be corrected by the time you look at the data.

I don’t wish suffering like this on anyone. GrabPERF had it’s own 3-4 day outage a few months ago. It’s just sad to see when monitoring services go down.

Last night, I got motivated.

Ok, I got manic. Goes with my life.

As a part of that mania, I had a breakthrough on how to present GrabPERF data that I’ve actually been collecting for nearly a year: text match failures.

GrabPERF has the ability to match text on page results using a standard PERL regex. By putting a regex into the measurement configuration, I can confirm that the data downloaded matches what should be there.

If there is not a match, the headers and page text are captured and inserted into a table. Up until yesterday, I was the only one who could view the data. Now, if you go to the graph configuration page (http://grabperf.org/measure_page.php?test=[insert test id here]), and see the following type of result, then click through the links.

If your graph configuration page says no text match configured, and you want one, let me know!

If you use GrabPERF on a regular basis, the somewhat flaky navigation method has become second nature to you. In fact, to circumvent some of the idiosyncrasies, you have probably bookmarked your favourite pages.

Yesterday, I broke your links.

When I redesigned GrabPERF in February 2006, I had just discover the require function in PHP, and decided to build the entire the structure using a single container page as the framework, and individual functions called using URL parameters.

As time went on, my own “brilliance” started to get in the way of maintaining and updating the code. It took me 10-15 minutes to figure out how I constructed pages, and then find the right code to fix or update.

Yesterday, I got completely fed up with this structure.

Now, all functions have their own unique pages, making maintenance a snap. And as an added benefit, I can now effectively track the usage of individual pages, so I know where to through development efforts.

Some of the changes.

http://grabperf.org/homepage.php?page=compare&test=2&tests%5B%5D=276&tests%5B%5D=277&tests%5B%5D=279&tests%5B%5D=280

becomes

http://grabperf.org/compare.php?test=2&tests%5B%5D=276&tests%5B%5D=277&tests%5B%5D=279&tests%5B%5D=280

http://grabperf.org/homepage.php?page=scatter&test=277&hours=2

becomes

http://grabperf.org/scatter.php?test=277&hours=2

I apologize for the confusion that this may cause, but in the long run, this will help me make the code better, and more robust.