For a number of years, I have owned three very popular domain names: WebPerformance.org, WebCaching.org, and WebCompression.org. Last night, after many days of consideration, I stopped pointing them at their own distinct Web space and pointed them at this blog.

This is not a bad or evil thing, considering that for at least 18 months, the articles that were hosted at those sites were duplicated here in a more manageable format.

For those who have come looking for the content from those sites, it is here. The search box in the right column can help you locate it.

But for those who would like a refresher, here is a list of the most popular articles on this blog, as selected by Web performance traffic.

Web Performance Concepts Series

• Web Performance, Part I: Fundamentals
• Web Performance, Part II: What are you calling ‘average’?
• Web Performance, Part III: Moving Beyond Average
• Web Performance, Part IV: Finding The Frequency
• Web Performance, Part V: Baseline Your Data
• Web Performance, Part VI: Benchmarking Your Site
• Web Performance, Part VII: Reliability and Consistency
• Web Performance, Part VIII: How do you define fast?
• Web Performance, Part IX: Curse of the Single Metric

Why Web Measurement Series

• Why Web Measurement, Part I: Customer Generation
• Why Web Measurement, Part II: Customer Retention
• Why Web Measurement, Part III: Business Operations
• Why Web Measurement, Part IV: Technical Operations

Web Performance Tuning

• The Dichotomy of the Web: Andy King’s Website Optimization
• Performance Improvement From Compression
• Baseline Testing With cURL
• Compressing Web Output Using mod_deflate and Apache 2.0.x
• Compressing PHP Output
• Using Client-Side Cache Solutions And Server-Side Caching Configurations To Improve Internet Performance
• Performance Improvement From Caching and Compression
• Compressing Web Output Using mod_gzip for Apache 1.3.x and 2.0.x
• mod_gzip Compile Instructions
• Hacking mod_deflate for Apache 2.0.44 and lower

IP Tracking

• Geographic IP database using PERL, PHP and MySQL — UPDATE: September 16 2008
• Hit Tracking with PHP and MySQL

Steven Hodson of WinExtra posted a screenshot of his personal Wordpress stats for the last three years last night. I then posted my stats for a similar period of time, and Steven shot back with some question about traffic, and the ebbs and flows of readers.

Being the stats nut that I am, I went and pulled the data from my own tracking data, and came up with this.

I made a conscious choice to analyze what year the posts being read were posted in. I wanted to understand when people read my content, which content kept people coming back over and over again. The chart above speaks for itself: through most of the last year it’s clear that the most popular posts were made in 2005.

What is also interesting is the decreasing interest in 2007 posts as 2008 progressed. Posts from 2006 remained steady, as there are a number of posts in that year that amount to my self-help guides to Web compression, mod_gzip, mod_deflate, and Web caching for Web administrators.

This data is no surprise to me, as I posted my rants against Gutter Helmet and their installation process in 2005. Those posts are still near the top of the Google search response for term “Gutter Helmet”. And improving the performance of a Web site is of great interest to many Apache server admins and Web site designers.

What is also clear is that self-hosting my blog and the posting renaissance it has provoked has driven traffic back to my site.

So, what lessons did I learn from this data?

1. Always remember the long tail. Every blogger wants to be relevant, on the edge, and showing that they understand current trends. The people who follow those trends are a small minority of the people who read blogs. Google and other search engines will expose them to your writings in the time of their choosing, and you may find that the three year-old post gets as much traffic as the one posted three hours ago
2. Write often. I was in a blogging funk when my blog was at Wordpress.com. As a geek, I believe that the lack of direct control over the look and feel of my content was the cause of this. In a self-hosted environment, I feel thta I am truly the one in charge, and I can make this blog what I want.
3. Be cautious of your fame. If your posts are front-loaded, i.e. if all your readers read posts from the month and year they are posted in, are you holding people’s long-term attention? What have you contributed to the ongoing needs of those who are outside the technical elite? What will drive them to keep coming to your site in the long run?

So, I post a challenge to other bloggers out there. My numbers are miniscule compared to the blogging elite, but I am curious to get a rough sense of how the long tail is treating you.

So, the last post I put up simply stated that I had failed in installing Worpress 2.6 on my own server. The permalinks were messed up and it looked a lot like an issue that a lot of people were having.

But, the catch was: the problem persisted when I tried to use Wordpress 2.5.1. Hmmm…wonder what’s up.

So today, I started thinking that it might be an Apache issue, rather than a Wordpress issue. So, taking a break from a massive project I’m doing (only so much performance data you can process before brain freeze), I started to poke around.

AllowOverride All

That did it. I was not letting Apache read the damn .htaccess file that Wordpress installed.

Excuse the thudding of my head on the desk.

Many apologies to the Automattic team. This user haz th dumb.

Dear Apache Software Foundation, and the developers of the Apache Web server:

I would like to thank you for developing a great product. I rely on it daily to host my own sites, and a large number of people on the Internet seem to share my love of this software.

However, it appears that you seem to want to maintain a simple flaw in your logic that continues to make me crazy. I am a Web performance analyst, and at least once a week I sigh, and shake my head whenever I stoop to use Microsoft Internet Explorer (MSIE) to visit secure sites.

I seems that in your SSL configurations, you continue to assume that ALL versions of MSIE can’t handle persistent connections under SSL/TLS.

Is this true? Is a bug initially caught in MSIE 5.x (5.0??) still valid for MSIE 6.0/7.0?

The short answer is: I don’t know.

It seems that no one in the Apache server team has bothered to go back and see if the current versions of MSIE — we are trying to track down the last three people use MSIE 5.x and help them — still share this problem.

In the meantime, can you change your SSL exclusion RegEx to something more, relevant for 2007?

Current RegEx:

SetEnvIf User-Agent ".*MSIE.*" nokeepalive
	ssl-unclean-shutdown
	downgrade-1.0 force-response-1.0

Relvant, updated REGEX:

SetEnvIf User-Agent ".*MSIE [1-5].*"
	nokeepalive ssl-unclean-shutdown
	downgrade-1.0 force-response-1.0

SetEnvIf User-Agent ".*MSIE [6-9].*"
	ssl-unclean-shutdown

Please? PLEASE? It’s so easy…and would solve so many performance problems…

Please?

Thank you.

Tags: Apache, SSL, TLS, MSIE, persistent connections, keepalive, web performance

It’s always funny when somewhat tech-savvy folks purposely make their bandwidth bills higher than they need to be.

Here’s TechCrunch’s HTTP header response.

HTTP/1.1 200 OK
Date: Tue, 16 Jan 2007 16:02:23 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 PHP/5.2.0-8 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8
X-Pingback: http://www.techcrunch.com/xmlrpc.php
Status: 200 OK
Transfer-Encoding: chunked
Content-Type: text/html; charset="UTF-8"

Port80 Software’s Compression Checker gives us some idea how much bandwidth Mr. Arrington, et al. could save just by activating this little feature, which comes baked into Apache 2.2.x.

Turn. On. Mod_deflate.

Port80 Software is reporting that in their survey of Fortune 1000 Web sites, IIS 6.0 has overtaken Apache as the Web server platform of choice. [here]

My two-cents: I respect the Port80 Software team greatly and love their maniacal devotion to ensuring that IIS users actually make use of the HTTP compression and caching that can so greatly improve Web performance.

That said, they are tied to Microsoft and the IIS platform. I would be curious to see if, scratching below the surface, they were able to determine what the application platform these companies built their mission critical Web applications on. I am open-minded and willing to hear that IIS is winning in that area as well. In my mind, it’s about Web performance tuning, not what you use to get that performance.

That said, I think a critical Web application survey of these same firms would find that many of these companies rely on JSP servers to run their core business processes.

As well, it would be interesting to se, by Fortune 1000 ranking, what the companies are using what server platform.

And…people still use Netscape Enterprise, SunOne, and Domino as production Web servers? YIKES!

This paper is an extension of the work done for another article that highlighted the performance benefits of retrieving uncompressed and compressed objects directly from the origin server. I wanted to add a proxy server into the stream and determine if proxy servers helped improve the performance of object downloads, and by how much.

Using the same series of objects in the original compression article[1], the CURL tests were re-run 3 times:

1. Directly from the origin server
2. Through the proxy server, to load the files into cache
3. Through the proxy server, to avoid retrieving files from the origin.[2]

This series of three tests was repeated twice: once for the uncompressed files, and then for the compressed objects.[3]

As can be seen clearly in the plots below, compression caused web page download times to improve greatly, when the objects were retrieved from the source. However, the performance difference between compressed and uncompressed data all but disappears when retrieving objects from a proxy server on a corporate LAN.

Instead of the linear growth between object size and download time seen in both of the retrieval tests that used the origin server (Source and Proxy Load data), the Proxy Draw data clearly shows the benefits that accrue when a proxy server is added to a network to assist with serving HTTP traffic.

	MEAN DOWNLOAD TIME
Uncompressed Pages
Total Time Uncompressed — No Proxy	0.256
Total Time Uncompressed — Proxy Load	0.254
Total Time Uncompressed — Proxy Draw	0.110
Compressed Pages
Total Time Compressed — No Proxy	0.181
Total Time Compressed — Proxy Load	0.140
Total Time Compressed — Proxy Draw	0.104

The data above shows just how much of an improvement is gained by adding a local proxy server, explicit caching descriptions and compression can add to a Web site. For sites that do force a great of requests to be returned directly to the origin server, compression will be of great help in reducing bandwidth costs and improving performance. However, by allowing pages to be cached in local proxy servers, the difference between compressed and uncompressed pages vanishes.

Conclusion

Compression is a very good start when attempting to optimize performance. The addition of explicit caching messages in server responses which allow proxy servers to serve cached data to clients on remote local LANs can improve performance to even a greater extent than compression can. These two should be used together to improve the overall performance of Web sites.

[1]The test set was made up of the 1952 HTML files located in the top directory of the Linux Documentation Project HTML archive.

[2]All of the pages in these tests announced the following server response header indicating its cacheability:

Cache-Control: max-age=3600

[3]A note on the compressed files: all compression was performed dynamically by mod_gzip for Apache/1.3.27.

How much improvement can you see with compression? The difference in measured download times on a very lightly loaded server indicates that the time to download the Base Page (the initial HTML file) improved by between 1.3 and 1.6 seconds across a very slow connection when compression was used.

Base Page Performance

There is a slightly slower time for the server to respond to a client requesting a compressed page. Measurements show that the median response time for the server averaged 0.23 seconds for the uncompressed page and 0.27 seconds for the compressed page. However, most Web server administrators should be willing to accept a 0.04 increase in response time to achieve a 1.5 second improvement in file transfer time.

First Byte Performance

Web pages are not completely HTML. How do improved HTML (and CSS) download times affect overall performance? The graph below shows that overall download times for the test page were 1 to 1.5 seconds better when the HTML files were compressed.

Total Page Performance

To further emphasize the value of compression, I ran a test on a Web server to see what the average compression ratio would be when requesting a very large number of files. As well, I wanted to determine what the affect on server response time would be when requesting large numbers of compressed files simultaneously. There were 1952 HTML files in the test directory and I checked the results using CURL across my local LAN.[1]

Large sample of File Requests (1952 HTML Files)

mod_gzip

mod_deflate[2]

Average Compression by File Size mod_gzip mod_deflate 0-999 0.713 0.777[3] 1000-4999 0.440 0.440 5000-9999 0.389 0.389 10000-19999 0.369 0.369 20000-49999 0.350 0.350 50000 and up 0.329 0.331

Conclusion

With a few simple commands, and a little bit of configuration, an Apache Web server can be configured to deliver a large amount of content in a compressed format. These benefits are not simply limited to static pages; dynamic pages generated by PHP and other dynamic content generators can be compressed by using the Apache compression modules. When added other performance tuning mechanisms and appropriate server-side caching rules, these modules can substantially reduce the bandwidth for a very low cost.

[1] The files were the top level HTML files from the Linux Documentation Project. They were installed on an Apache 1.3.27 server running mod_gzip and an Apache 2.0.44 server using mod_deflate. Minimum file size was 80 bytes and maximum file size was 99419 bytes.

[2] mod_deflate for Apache/2.0.44 and earlier comes with the compression ratio set for Best Speed, not Best Compression. This configuration can be modified using the tips found here; and starting with Apache/2.0.45, there will be a configuration directive that will allow admins to configure the compression ratio that they want.

In this example, the compression ratio was set to Level 6.

[3] mod_deflate does not have a lower bound for file size, so it attempts to compress files that are too small to benefit from compression. This results in files smaller than approximately 120 bytes becoming larger when processed by mod_deflate.

The last time I attempted to compile mod_gzip into Apache, I found that the instructions for doing so were not documented clearly on the project page. After a couple of failed attempts, I finally found the instructions buried at the end of the ChangeLog document.

I present the instructions here to preserve your sanity.

Before you can actually get mod_gzip to work, you have to uncomment it in the httpd.conf file module list (Apache 1.3.x) or add it to the module list (Apache 2.0.x).

Now there are two ways to build mod_gzip: statically compiled into Apache and a DSO-File for mod_so. If you want to compile it statically into Apache, just copy the source to Apache src/modules directory and there into a subdirectory named ‘gzip’. You can activate it via a parameter of the configure script.

 ./configure --activate-module=src/modules/gzip/mod_gzip.a
 make
 make install

This will build a new Apache with mod_gzip statically built in.

The DSO-Version is much easier to build.

 make APXS=/path/to/apxs
 make install APXS=/path/to/apxs
 /path/to/apachectl graceful

The apxs script is normally located inside the bin directory of Apache.

NOTE: This hack is only relevant to Apache 2.0.44 or lower. Starting with Apache 2.0.45, the server contains the DeflateCompressionLevel directive, which allows for user-configured compression levels in the httpd.conf file.

One of the complaints leveled against mod_deflate for Apache 2.0.44 and below has been the lower compression ratio that it produces when compared to mod_gzip for Apache 1.3.x and 2.0.x. This issue has been traced to a decision made by the author of mod_deflate to focus on fast compression versus maximum compression.

In discussions with the author of mod_deflate and the maintainer of mod_gzip, the location of the issue was quickly found. The level of compression can be easily modified by changing the ZLIB compression setting in mod_deflate.c from Z_BEST_SPEED (equivalent to “zip -1″) to Z_BEST_COMPRESSION (equivalent to “zip -9″). These defaults can also be replaced with a numeric value between 1 and 9. A “hacked” version of the mod_deflate.c code is available here. In this file, the compression level has been set to 6, which is regarded as a good balance between speed and compression (and also happens to be ZLIB’s default ratio). Some other variations are highlighted below.

zRC = deflateInit2(&ctx->stream, Z_BEST_SPEED, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

1. zRC = deflateInit2(&ctx->stream, Z_BEST_COMPRESSION, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

2. zRC = deflateInit2(&ctx->stream, 6, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

3. zRC = deflateInit2(&ctx->stream, 9, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);