Since I started self-hosting this blog again on August 6 2008, I have been trying to find more ways to pull traffic toward the content that I put up. Like all bloggers, I feel that I have important things to say (at least in the area of Web performance), and ideas that should be read by as many people as possible.

As well, I have realized that if I invest some time and effort into this blog, it can be a small revenue source that could get me that much closer to my dream of a MacBook Pro.

The Analysis

In a post yesterday morning, Darren Rowse had some advice on when the best time to release new post is. Using his ideas as the framework, I pulled the data out of my own tracking database and came up with the chart below. This shows the page view data between September 1 2007 and September 15 2008 based on the day of the week vistors came to the site.

Using this data and the general framework that Darren subscribes to, I should be releasing my best and newest thoughts in a week on Monday and Tuesday (GMT).

After Wednesday, I should release only less in-depth articles, with a focus on commentary on news and events. And I must learn to breathe, as I suffer from an ailment all to common in bipolars: a lack of patience.

A new post doesn’t immediately find its target audience unless you have hundreds or thousands (Tens? Ones?) of readers who are influential. If you are luckyin this regard, then these folks will leave useful comments, and through their own attention, help gently show people that a new post is something they should devote their valuable attention towards.

It takes a while for any post to percolate through the intertubes. So patience you must have.

Front-loaded v Long-tailed

Unless, of course, your traffic model is completely different than a popular blogger.

The one issue that I had with Darren’s guidance is that it applies only to blogs that are front-loaded. A front-loaded blog is one that is incredibly popular, or has a devoted, active audience who help push page views toward the most recent 3-5 posts. Once the wave has crested, or the blogger has posted something new, the volume of traffic to older posts falls off exponentially, except in the few cases of profound or controversial topics.

When I analyzed my own traffic, I found that the most of my traffic volume was aimed toward posts from 2005 and 2006. In fact, more recent posts are nowhere near as popular as these older posts. In contrast to the front-loaded blog, mine is long-tailed.

There are a number of influential items in my blog which have proven staying power, which draw people from around the world. They have had deep penetration into search engines, and are relvant to some aspect of peoples’ lives that keeps pulling them back.

Summary

I would highly recommend analyzing your traffic to see it is front-loaded or long-tailed. I know that I wish that this blog  was more front-loaded, with an active community of readers and commentators. However, I am also happy to see that I have created a few sparks of content that keep people returning again and again. If your blog is  long-tailed, then when you post becomes far less relevant than ensuring the freshness and validity of those few popular posts. Ensure that these are maintained and current so that they remain relevant to as many people as possible.

Recently there was an outage at a hit-tracking vendor I was using to track the hits on my externally hosted blog, leaving me with a gap in my visitor data several hours long. While this was an inconvenience for me, I realized that this could be mission critical failure to an online business reliant on this data.

To resolve this, I used the PHP HTTP environment variables and the built-in function for converting IP addresses to IP numbers to create my own hit-tracker. It is a rudimentary tracking tool, but it provides me with the basic information I need to track visitors.

To begin, I wrote a simple PHP script to insert tracking data into a MySQL database. How do you do that? You use the gd features in PHP to draw an image, and insert the data into the database.

header ("Content-type: image/png");

include("dbconnect_logger.php");
$logtime = date("YmdHis");
$ipquery = sprintf("%u",ip2long($_SERVER['REMOTE_ADDR']));

        $query2 = "INSERT into logger.blog_log values \
               ($logtime,$ipquery,'$HTTP_USER_AGENT','$HTTP_REFERER')";
        mysql_query($query2) or die("Log Insert Failed");

mysql_close($link);

$im = @ImageCreate (1, 1)
or die ("Cannot Initialize new GD image stream");
$background_color = ImageColorAllocate ($im, 224, 234, 234);
$text_color = ImageColorAllocate ($im, 233, 14, 91);

// imageline ($im,$x1,$y1,$x2,$y2,$text_color);
imageline ($im,0,0,1,2,$text_color);
imageline ($im,1,0,0,2,$text_color);

ImagePng ($im);
?>

DROP TABLE IF EXISTS `blog_log`;
CREATE TABLE `blog_log` (
  `date` timestamp NOT NULL default '0000-00-00 00:00:00',
  `ip_num` double NOT NULL default '0',
  `uagent` varchar(200) default NULL,
  `visited_page` varchar(200) NOT NULL default '',
  UNIQUE KEY `date` (`date`,`ip_num`,`visited_page`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Data should begin flowing to your database immediately. This sample snippet of code will allow you to pull data for a selected day and list each individual hit.

$query1 = "SELECT
                bl.ip_num,
                DATE_FORMAT(bl.date,'%d/%b/%Y %H:%i:%s') AS NEW_DATE,
                bl.uagent,
                bl.visited_page
        FROM blog_log bl
        WHERE
                DATE_FORMAT(bl.date,'%Y%m%d') ='$YMD'
		and uagent not REGEXP '(.*bot.*|.*crawl.*|.*spider.*|^-$|.*slurp.*|.*walker.*|.*lwp.*|.*teoma.*|.*aggregator.*|.*reader.*|.*libwww.*)'
        ORDER BY bl.date ASC";

print "<table border=\"1\">\n";
print "<tr><td>IP</td><td>DATE</td><td>USER-AGENT</td><td>PAGE VIEWED</td></tr>";
while ($row = mysql_fetch_array($result1)) {
        $visitor = long2ip($row[ip_num]);
        print "<tr><td>$visitor</td><td nowrap>$row[NEW_DATE]</td><td nowrap>$row[uagent]</td><td>";

	if ($row[visited_page] == ""){
    	    print " --- </td></tr>\n";
	} else {
    	    print "<a href=\"$row[visited_page]\" target=\_blank\">$row[visited_page]</a></td></tr>\n";
	}

}

mysql_close($link);

For my own purposes, this is an extension of the Geographic IP database I created a number of years ago. This application extracts IP address information from the five IP registrars, and inserts it into a database. Using the log data collected by the tracking bug above and the lookup capabilities of the Geographic IP database, I can quickly track which countries and ISP drive the most visitors to my site, and use this for general interest purposes, as well as the ability to isolate any malicious visitors to the site.

Since I migrated the blog back to my own servers a few days ago, I have realized something: I have fallen back to my old habit of watching the hit count.

This is weird, considering the lack of interest I had in my blog and its stats over the last year or so. But having my baby back home where I am in charge gives a sense that I should pay attention. That I need to know what’s happening.

In 2005, when I started doing this, I used to watch my hit count religiously, maniacally. Sort of goes with the bipolar, but I digress. In 2008, we are obsessed with followers, and the Slashdot like addiction to being the first to report some breaking (planted?) news item.

So, after three years of blogging, I see that the online communities haven’t progressed much beyond hit counters, page views, or followers. And online cred is a insular and self-perpetuating thing. You draw attention to yourself, you get comments, more people follow you, and more people feed you, you have more to say, more people follow you…and on, and on.

I am not saying that this is good, or ill. I am as much a traffic whore as the rest of the world. I just realize that we are all after the same goal – attention. That was the whole idea behind the Attention Economy, a term I don’t hear as much as I did 2 years ago.

With FriendFeed and Twitter, we live in the Attention Economy. With 200 channels in my basic cable package, TV is a passive Attention Economy, controlled by the PVR and the TorrentSphere. Satellite Radio forces us to make choices.

Be it hit-counts or PVRs, we all crave attention, knowing full well how limited the attention-span is. We don’t want be to waste our time, but we want to attract that of others.

Attention produces an unbalanced online economy. We can do many things to control the incoming flow. We can also work very hard to expand the outgoing flow. But for most of us, the outgoing flow remains a trickle, maybe even a fine mist.

So where does the power in the Attention Economy lie? With the off-switch.

And we all have one.

How do you use yours?

One of the performance hits that the GrabPERF system has is the dynamic generation of the main page. The nature of the SQL calls and the underlying PHP makes it scale exponentially past a certain number of measurements.

Last night, Kevin Burton made a grand suggestion: generate a static page on a regular schedule.

Duh!

Today, I wrote the script that does this. The performance of the main page has adjusted accordingly.

Yikes!

UPDATE: Ian Holsman reminded that if I use cURL, I can use the exiting PHP to build the pages without a PERL script.

I. AM. AN. IDIOT.

Now, bedtime.

Technorati Tags: GrabPERF, Web+performance, PHP, MySQL, Dynamic+pages, Static+pages, HTML

Over the last year, GrabPERF has been something that has caught the fancy of a few in the Blogging/Social Media world. It has given some perspective of how performance can affect business and image in the connected world.

But what of GrabPERF itself? It has been on a development hiatus for the last few months due to pressures from my “real” job and various trips (business and pleasure) that I have been undertaking. Over the last two weeks, I have been trying to clear out the extra measurements and focus the features and attention on the community that appears most interested in the data.

During this process, I heard back from some folks who had been using GrabPERF in stealth mode (even I can’t track all the hits!), and who asked, “Hey! Where did my data go?”. Glad to hear from all of you.

Just to give everyone some idea of the growth, here is a snapshot of aggregated daily performance and number of measurements.

The number of measurements shot up, until I started culling the unused measurements. Over the last 3 weeks, average performance became extremely variable, and that’s when I began considering the culling. As well, the New York PubSub Agent appears to have gone permanently offline, as a part of their winding down process.

The fact that the system was taking 390,000 measurements per day still astounds me.

This was also comparable to the number of distinct sites we were measuring.

After the latest cull, we are down to 84 distinct tests, a level last seen on November 27, 2005.

I am pleased that the system has held together as well as it has.

Technorati Tags: GrabPERF, statistics

Rick Segal has a great post this morning about the unique nature of the Canada-US border. [here]

Rick hits it on the head: US Customs and Immigration Agents are some of the most unpleasant people I have ever had the pleasure of dealing with.

When Samantha goes home with the boys next month, what does she fear the most?

Re-entering the US on a valid visa.

Why should people who have valid visas fear coming back into the US?

Oh yeah, the US hates immigrants. Or at least that’s how the rest of the world interprets their attitude, policies, and actions toward those who are not privileged enough to hold US citizenship.

UPDATE: Looks like Matt Mullenweg enjoyed the pleasure of what happens going INTO Canada without a passport! [here]

UPDATE: GOOD LORD! Is it weird immigration story day or what? David Weinberger recounts his encounter with a US Immigration Agent in Montreal last night. Gives me some inkling of hope.

Technorati Tags: DHS, US Customs, Canada Customs, Canada, US, immigration, work visa, h-1b

The update on my Gutter Helmet posts (1 and 2) is that there is no update.

We received a phone call from a local Gutter Helmet rep about 2 weeks ago. He spoke to Samantha who, outlined some of our concerns and issues. I then called and left him a message last week, re-iterating these same concerns, and noting that there was still a leak between the new roof and the new gutters which needed to be fixed by flashing.

Nothing. No response. Silence.

I know it’s Gutter Helmet’s busy season. I know this because the number of hits to my previous posts are increasing.

Maybe some customers are experiencing better installations; I hope so. All I can do is continue to recount my experience to you.

IceRocket: Gutter Helmet, gutters, contractors, customer service

Seems that us Introverts are busting out everywhere.

Jonathan Rauch hits for six with this article.

I work at a company run by Extroverts, who, without a second thought, promote the Extroverts and marginalize the Introverts.

The thing is that in my company, outside of sales, the company is dominated by Introverts. In fact, because it is a technology company, the Introverts are really the ones who have the power.

And the Extroverts wonder why the Introverts DESPISE the open plan office — no cubes, no offices.

Where are the companies run by Introverts, for Introverts?

Zane hits for six with this commentary.

Why? Why does this nation do these things? What makes people simply shrug their shoulders and say, “oh well; thankfully it’s not me”.

Next time, it will be you. Or someone you know. And what then?

Why does this exchange bother me?

[ISP NAME DELETED]:

I have been seeing a VERY large number of visits from two hosts in
your IP block since August 26, 2005.

A list of all of the hits since Aug 26 is attached.

If you do not stop this/these hosts, I will be forced to drop all traffic from the [/24 IP block deleted] IP block.

Thank you for your attention to this matter.

Their response:

Best thing to do is drop the traffic.

Truly Yours,
[ISP NAME DELETED] Abuse Team

Ummmm, WTF? Did I miss something?