Since I started self-hosting this blog again on August 6 2008, I have been trying to find more ways to pull traffic toward the content that I put up. Like all bloggers, I feel that I have important things to say (at least in the area of Web performance), and ideas that should be read by as many people as possible.

As well, I have realized that if I invest some time and effort into this blog, it can be a small revenue source that could get me that much closer to my dream of a MacBook Pro.

The Analysis

In a post yesterday morning, Darren Rowse had some advice on when the best time to release new post is. Using his ideas as the framework, I pulled the data out of my own tracking database and came up with the chart below. This shows the page view data between September 1 2007 and September 15 2008 based on the day of the week vistors came to the site.

Using this data and the general framework that Darren subscribes to, I should be releasing my best and newest thoughts in a week on Monday and Tuesday (GMT).

After Wednesday, I should release only less in-depth articles, with a focus on commentary on news and events. And I must learn to breathe, as I suffer from an ailment all to common in bipolars: a lack of patience.

A new post doesn’t immediately find its target audience unless you have hundreds or thousands (Tens? Ones?) of readers who are influential. If you are luckyin this regard, then these folks will leave useful comments, and through their own attention, help gently show people that a new post is something they should devote their valuable attention towards.

It takes a while for any post to percolate through the intertubes. So patience you must have.

Front-loaded v Long-tailed

Unless, of course, your traffic model is completely different than a popular blogger.

The one issue that I had with Darren’s guidance is that it applies only to blogs that are front-loaded. A front-loaded blog is one that is incredibly popular, or has a devoted, active audience who help push page views toward the most recent 3-5 posts. Once the wave has crested, or the blogger has posted something new, the volume of traffic to older posts falls off exponentially, except in the few cases of profound or controversial topics.

When I analyzed my own traffic, I found that the most of my traffic volume was aimed toward posts from 2005 and 2006. In fact, more recent posts are nowhere near as popular as these older posts. In contrast to the front-loaded blog, mine is long-tailed.

There are a number of influential items in my blog which have proven staying power, which draw people from around the world. They have had deep penetration into search engines, and are relvant to some aspect of peoples’ lives that keeps pulling them back.

Summary

I would highly recommend analyzing your traffic to see it is front-loaded or long-tailed. I know that I wish that this blog  was more front-loaded, with an active community of readers and commentators. However, I am also happy to see that I have created a few sparks of content that keep people returning again and again. If your blog is  long-tailed, then when you post becomes far less relevant than ensuring the freshness and validity of those few popular posts. Ensure that these are maintained and current so that they remain relevant to as many people as possible.

If you use GrabPERF on a regular basis, the somewhat flaky navigation method has become second nature to you. In fact, to circumvent some of the idiosyncrasies, you have probably bookmarked your favourite pages.

Yesterday, I broke your links.

When I redesigned GrabPERF in February 2006, I had just discover the require function in PHP, and decided to build the entire the structure using a single container page as the framework, and individual functions called using URL parameters.

As time went on, my own “brilliance” started to get in the way of maintaining and updating the code. It took me 10-15 minutes to figure out how I constructed pages, and then find the right code to fix or update.

Yesterday, I got completely fed up with this structure.

Now, all functions have their own unique pages, making maintenance a snap. And as an added benefit, I can now effectively track the usage of individual pages, so I know where to through development efforts.

Some of the changes.

http://grabperf.org/homepage.php?page=compare&test=2&tests%5B%5D=276&tests%5B%5D=277&tests%5B%5D=279&tests%5B%5D=280

becomes

http://grabperf.org/compare.php?test=2&tests%5B%5D=276&tests%5B%5D=277&tests%5B%5D=279&tests%5B%5D=280

http://grabperf.org/homepage.php?page=scatter&test=277&hours=2

becomes

http://grabperf.org/scatter.php?test=277&hours=2

I apologize for the confusion that this may cause, but in the long run, this will help me make the code better, and more robust.

Lifehack.org lists the top ten reasons to being an Introvert. As an INTJ, I can relate to these.

10) Work Well With Others, Especially In One-to-One Relationships
9) Maintain Long-Term Friendships Flexible
7) Independent
6) Strong Ability To Concentrate
5) Self-Reflective
4) Responsible
3) Creative, Out-of-the-Box thinking
2) Analytical Skills That Integrate Complexity
1) Studious and Smart

Looks like I need to buy The Introvert Advantage.

Ok, one of the tests that the GrabPERF System is running is doing a simple search on the Technorati site.

Ouch.

Now, as I mentioned before, Technorati has some interesting things going on in their www servers. For the Web geeks out there, here is what their headers look like.

HTTP/1.0 200 OK
Date: Mon, 11 Jul 2005 16:15:47 GMT
Server: Apache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
X-Powered-By: PHP/4.3.11
Set-Cookie: TECHNORATI_MEMBER=deleted; \
                   expires=Sun, 11 Jul 2004 16:15:46 GMT; path=/
Set-Cookie: TECHNORATI_MEMBER=[blah-blah]; \
                   expires=Wed, 12 Oct 2005 16:15:47 GMT; path=/; \
                   domain=technorati.com
Pragma: no-cache
Content-Type: text/html; charset=UTF-8
Connection: close

The Cache-Control header I can understand. The Pragma: no-cache is basically useless, as the Pragma header is only considered as a valid client-side header.

But the use of HTTP/1.0 and the explicit Connection: close really bother me. Removing the ability for clients to maintain persistent connections is a Network and Server resource issue, and should be avoided at all costs.

And where’s the compression? There is no need to send raw content to clients that understand and can process compressing text content. Using either hardware or mod_gzip or mod_deflate, the size of transferred content can be very easily reduced.

Sorry for the rant, but I seem to be on a Web performance kick lately. And the team at Technorati is one that I would expect to have gotten the need for this Web performance thing.

Technorati: web performance, HTTP, Technorati

Rick Segal and I have different, if not complementary perspectives on the United States. He is a US citizen living in Canada; I am a Canadian living in the US. I have met Rick in person, and he is an intelligent and insightful person to speak with.

Last night, he posted his reaction to the US Senate’s lack of unanimous support for the apology for lynching. [here]

This story hits home for me as well. I have chosen to live in this country. I know that as a guest, I should be quiet and not rock the boat. But when a situation such as this arises, along with the increasing levels of spitfire rhetoric and poisonous debate, I have to step back and ask: why?

For the most part, my life in the US has been one of comfort and ease. However, when I moved to the US in 1999, I was surprised at just how different two countries could be who looked so much alike.

Last year’s presidential campaign, the second one I have been through while here, further highlighted that the voices of moderation and rational thought are lost in the need to polarize and inflame.

But, as Rick says, at 32,000 feet, all of that is gone. There aren’t countries; or red states and blue states. Just the ever morphing tapestry of the world we live on. The hand of man can be seen from this height. Nature is still in control, shaping how man shapes the land.

We will come and go. We all die. But our legacy will be remembered, and, increasingly, recorded for all posterity.

Be true to yourself. Remember, the way history judges you is beyond your control.

David Janes is maintaining a list of the non-sponsoring Senators.

Went in to urgent care today — I have a sinus infection.

But I saw a new doctor, and I liked her a lot. And I figured out why.

She is a doctor. She doesn’t ACT like a doctor. My regular doctor always wears a white coat and maintains a cold, impersonal approach. The doctor I saw today was a human being, and treated me like a person.

A good rule for anyone: be yourself, not the role you think you should play (unless you are an actor).

The Dean of the School of Library and Information Sciences at Indiana University, Blaise Cronin, maintains an open mind about blogging. [here]

One wonders for whom these hapless souls blog. Why do they choose to expose their unremarkable opinions, sententious drivel and unedifying private lives to the potential gaze of total strangers? What prompts this particular kind of digital exhibitionism? The present generation of bloggers seems to imagine that such crassly egotistical behavior is socially acceptable and that time-honored editorial and filtering functions have no place in cyberspace. Undoubtedly, these are the same individuals who believe that the free-for-all, communitarian approach of Wikipedia is the way forward. Librarians, of course, know better.

Books are great. I still love to handle a book or magazine. But almost everything I have learned in the last 2 years has come from the Web. I have learned more about Sales and Marketing through blogs than any MBA program. I have learned how to quickly and clearly communicate complex ideas by putting my ideas on the Web for commentary.

Libraries will always exist; attitudes like Dr. Cronin’s will not.

Via Kevin Briody

Kathy Sierra writes on the geography of context. The main takeaway: North Americans remove objects from their context; Asians always consider the whole picture.

A thought-provoking reminder of our differences as a species. I am defiitely guilty of removing objects from their context and looking at them in isolation. However, working with computer systems on a daily basis is one method that I find for helping maintain contextual grounding.

Ok…the above sentence was hideous. What I meant to say is that workig with complex Internet systems forces a person to always think of devices, applications and configurations in the context that they exist.

In fact, that is the main failing inside most of the organizations that I work with. There is no overarching understanding of how all of the systems work together.

So, the culture of context can exist within an organization. I gravitate towards people who are able to bring in the whole picture.

Who do you spend time with? Isolationists or Inclusionists?

Web page compression is not a new technology, but it has just recently gained higher recognition in the minds of IT administrators and managers because of the rapid ROI it generates. Compression extensions exist for most of the major Web server platforms, but in this article I will focus on the Apache and mod_gzip solution.

The idea behind GZIP-encoding documents is very straightforward. Take a file that is to be transmitted to a Web client, and send a compressed version of the data, rather than the raw file as it exists on the filesystem. Depending on the size of the file, the compressed version can run anywhere from 50% to 20% of the original file size.

In Apache, this can be achieved using a couple of different methods. Content Negotiation, which requires that two separate sets of HTML files be generated — one for clients that can handle GZIP-encoding, and one for those who can’t — is one method. The problem with this solution should be readily apparent: there is no provision in this methodology for GZIP-encoding dynamically-generated pages.

The more graceful solution for administrators who want to add GZIP-encoding to Apache is the use of mod_gzip. I consider it one of the overlooked gems for designing a high-performance Web server. Using this module, configured file types — based on file extension or MIME type — will be compressed using GZIP-encoding after they have been processed by all of Apache’s other modules, and before they are sent to the client. The compressed data that is generated reduces the number of bytes transferred to the client, without any loss in the structure or content of the original, uncompressed document.

mod_gzip can be compiled into Apache as either a static or dynamic module; I have chosen to compile it as a dynamic module in my own server (more compile instructions here). The advantage of using mod_gzip is that this method requires that nothing be done on the client side to make it work. All current browsers — Mozilla, Opera, and even Internet Explorer — understand and can process GZIP-encoded text content.

On the server side, all the server or site administrator has to do is compile the module, edit the appropriate configuration directives that were added to the httpd.conf file, enable the module in the httpd.conf file, and restart the server. In less than 10 minutes, you can be serving static and dynamic content using GZIP-encoding without the need to maintain multiple codebases for clients that can or cannot accept GZIP-encoded documents.

When a request is received from a client, Apache determines if mod_gzip should be invoked by noting if the “Accept-Encoding: gzip” HTTP request header has been sent by the client. If the client sends the header, mod_gzip will automatically compress the output of all configured file types when sending them to the client.

This client header announces to Apache that the client will understand files that have been GZIP-encoded. mod_gzip then processes the outgoing content and includes the following server response headers.

		Content-Type: text/html
		Content-Encoding: gzip
		

These server response headers announce that the content returned from the server is GZIP-encoded, but that when the content is expanded by the client application, it should be treated as a standard HTML file. Not only is this successful for static HTML files, but this can be applied to pages that contain dynamic elements, such as those produced by Server-Side Includes (SSI), PHP, and other dynamic page generation methods. You can also use it to compress your Cascading Stylesheets (CSS) and plain text files. As well, a whole range of application file types can be compressed and sent to clients. My httpd.conf file sets the following configuration for the file types handled by mod_gzip:

		mod_gzip_item_include mime ^text/.*
		mod_gzip_item_include mime ^application/postscript$
		mod_gzip_item_include mime ^application/ms.*$
		mod_gzip_item_include mime ^application/vnd.*$
		mod_gzip_item_exclude mime ^application/x-javascript$
		mod_gzip_item_exclude mime ^image/.*$
		

This allows Microsoft Office and Postscript files to be GZIP-encoded, while not affecting PDF files. PDF files should not be GZIP-encoded, as they are already compressed in their native format, and compressing them leads to issues when attempting to display the files in Adobe Acrobat Reader.^[1] For the paranoid system administrator, you may want to explicitly exclude PDF files.

		mod_gzip_item_exclude mime ^application/pdf$
		

Another side-note is that nothing needs to be done to allow the GZIP-encoding of OpenOffice (and presumably, StarOffice) documents. Their MIME-type is already set to text-plain, allowing them to be covered by one of the default rules.

How beneficial is sending GZIP-encoded content? In some simple tests I ran on my Web server using WGET, GZIP-encoded documents showed that even on a small Web server, there is the potential to produce a substantial savings in bandwidth usage.

Server administrators may be concerned that mod_gzip will place a heavy burden on their systems as files are compressed on the fly. I argue against that, pointing out that this does not seem to concern the administrators of Slashdot, one of the busiest Web servers on the Internet, who use mod_gzip in their very high-traffic environment.

The mod_gzip project page for Apache 1.3.x is located at SourceForge. The Apache 2.0.x version is available from here.

[1] From http://www.15seconds.com/issue/020314.htm

Back in November, I mentioned that I was working on the idea of new ways to benchmark the success of online businesses in today’s more mature operational environment. I am still working on the base ideas, but a colleague of mine has helped me coalesce some ideas, and they are now forming the foundation of the concepts my company will begin using internally to more deeply understand the various Web performance benchmarks we monitor.

For those who use the existing Web performance benchmarks to determine the success and failure of your online business, you understand how thin the veneer is on these benchmarks. They do not provide true insight into the operational success of an online business, and they are more likely to sow the seeds of distrust between IT and Business operations in the long-term by creating an artificial standard which becomes the goal.

If an online business truly wants to achieve and maintain exemplary Web performance numbers, it has to start with a strong foundation, and build on it. Why? The team I work with spends a lot of time trying to understand and reverse engineer the broken processes, designs, and architectures that were laid out in order to get big fast. After 3-4 years of technical starvation and underfunding, these online businesses are beginning to show strain; the temporary fix has become the permanently broken process.

The rush into the Web analytics space in the last few weeks is a key sign that companies now see value in and want to exploit the vast quantities of data that they collect on their traffic daily. Web analytics is an astoundingly complex field, but most people boil it up to a single concept: How many Unique Page Views did I get?

Unique Page Views is an outdated Web server analytics metric. It does not tell me anything about the business, other than it has a lot of traffic. Back in the “eyeballs are everything” period, this would have been a big deal. Now, I say so what, and start asking:

• From where
• Dialup? Broadband?
• How many were able to successfully complete their transactions?
• What paths are most visited
• Average spend by connection type?
• Average spend by hour?
• etc.

Like Unique Page Views, the average Web performance and availability of a Web page or transaction does not accurately represent the overall health of any online business. Within the large populations of data that exist at the Web measurement firms, there is a wealth of data that could be used to clearly expose more important benchmarking statistics.

If you are from an online business, you already understand that the average performance over an artificially-defined period of time is a very inaccurate way to measure the success of the online business. However, it is the accepted standard in the field. Underlying those aggregated values, there are clearly-defined statistical methods which can be used to extract even more meaningful information from the mass of measurement data.

I would discuss more of the ideas and concepts that we are working on, but I know that I do get visitors from our competitors, so I will have to keep our ideas under wraps for right now.

But I want to hear yours. What does your online business use as a benchmark for success? Standard avergae performance and availability? Or something more complex that examines the performance data as a complete population, as opposed to an aggregated summary value? Does your firm tie business goals and objectives into the performance benchmarks so that people across the company can understand how the business is succeeding, and how delivering a good, bad, and downright awful online performance experience can affect the bottom line?

This is an exciting time to have access to large amounts of data on the health of the Internet.