It’s always funny when somewhat tech-savvy folks purposely make their bandwidth bills higher than they need to be.

Here’s TechCrunch’s HTTP header response.

HTTP/1.1 200 OK
Date: Tue, 16 Jan 2007 16:02:23 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 PHP/5.2.0-8 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8
X-Pingback: http://www.techcrunch.com/xmlrpc.php
Status: 200 OK
Transfer-Encoding: chunked
Content-Type: text/html; charset="UTF-8"

Port80 Software’s Compression Checker gives us some idea how much bandwidth Mr. Arrington, et al. could save just by activating this little feature, which comes baked into Apache 2.2.x.

Turn. On. Mod_deflate.

It’s the places that you go when you’re a child that stay with you for your whole life.

My grandparents lived their entire lives in the Crowsnest Pass. This narrow, sometimes forgotten section of the Rockies emptied itself of its coal to feed the engines of Canada and the world for more than a hundred years.

My grandfathers, and my great-grandfathers, all gave their lives to the dirty work of ripping this black gold from the bowels of the earth. Their bodies showed the scars of a life lived in darkness, straining to pull themselves through another day.

When it got to much, they drank. They fought. They dreamed. Some escaped, some took their own lives, many just survived.

The Alberta side of the Pass — no one who has spent any time in the area ever uses “Crowsnest Pass” — is slowly dying. The generation who mined underground is dying away. The next generation, and the one after them, has taken to tearing the tops off mountains in BC.

Or they just left, like my parents did. They empty carcasses of a life abandoned for economics are still there.

I was back there this summer for the first time since 1999. It has come a long way, but their is an aura, a feeling that the end is near. All the money from Calgary can’t save them. The old, independent life, the hardened bitterness, the brutal economics of coal that bred a people that accepted all into the brotherhood of the black gold, is gone.

There was a bluff outside the Pass community of Coleman, full of what the locals called “black diamonds”. I’m not sure if it was jet (made from extra compression on some of the coal deposits), or obsidian (from the volcanic activity that dominated the area in previous epochs). Sometimes, if the light was right, you could see the light reflecting off the pieces showing through the bluff.

Then, about 15 years ago, in order to straighten the highway and let more huge trailer trucks roar through Coleman on their way to the rest of the world, the bluff was blown away.

Sometimes, in the rush of time, the memories in our blood get blown away, each individual event glistening in the sun one last time, before being scooped up and swept away.

Well, the irony of this is painful.

I went with 1&1 as the hosting location for my personal domains, including WebPerformance.org.

One of the things that I preach there is the use of compression.

Guess what? 1&1 doesn’t use Web compression on their servers.

Ugh.

Port80 Software is reporting that in their survey of Fortune 1000 Web sites, IIS 6.0 has overtaken Apache as the Web server platform of choice. [here]

My two-cents: I respect the Port80 Software team greatly and love their maniacal devotion to ensuring that IIS users actually make use of the HTTP compression and caching that can so greatly improve Web performance.

That said, they are tied to Microsoft and the IIS platform. I would be curious to see if, scratching below the surface, they were able to determine what the application platform these companies built their mission critical Web applications on. I am open-minded and willing to hear that IIS is winning in that area as well. In my mind, it’s about Web performance tuning, not what you use to get that performance.

That said, I think a critical Web application survey of these same firms would find that many of these companies rely on JSP servers to run their core business processes.

As well, it would be interesting to se, by Fortune 1000 ranking, what the companies are using what server platform.

And…people still use Netscape Enterprise, SunOne, and Domino as production Web servers? YIKES!

This paper is an extension of the work done for another article that highlighted the performance benefits of retrieving uncompressed and compressed objects directly from the origin server. I wanted to add a proxy server into the stream and determine if proxy servers helped improve the performance of object downloads, and by how much.

Using the same series of objects in the original compression article[1], the CURL tests were re-run 3 times:

1. Directly from the origin server
2. Through the proxy server, to load the files into cache
3. Through the proxy server, to avoid retrieving files from the origin.[2]

This series of three tests was repeated twice: once for the uncompressed files, and then for the compressed objects.[3]

As can be seen clearly in the plots below, compression caused web page download times to improve greatly, when the objects were retrieved from the source. However, the performance difference between compressed and uncompressed data all but disappears when retrieving objects from a proxy server on a corporate LAN.

Instead of the linear growth between object size and download time seen in both of the retrieval tests that used the origin server (Source and Proxy Load data), the Proxy Draw data clearly shows the benefits that accrue when a proxy server is added to a network to assist with serving HTTP traffic.

	MEAN DOWNLOAD TIME
Uncompressed Pages
Total Time Uncompressed — No Proxy	0.256
Total Time Uncompressed — Proxy Load	0.254
Total Time Uncompressed — Proxy Draw	0.110
Compressed Pages
Total Time Compressed — No Proxy	0.181
Total Time Compressed — Proxy Load	0.140
Total Time Compressed — Proxy Draw	0.104

The data above shows just how much of an improvement is gained by adding a local proxy server, explicit caching descriptions and compression can add to a Web site. For sites that do force a great of requests to be returned directly to the origin server, compression will be of great help in reducing bandwidth costs and improving performance. However, by allowing pages to be cached in local proxy servers, the difference between compressed and uncompressed pages vanishes.

Conclusion

Compression is a very good start when attempting to optimize performance. The addition of explicit caching messages in server responses which allow proxy servers to serve cached data to clients on remote local LANs can improve performance to even a greater extent than compression can. These two should be used together to improve the overall performance of Web sites.

[1]The test set was made up of the 1952 HTML files located in the top directory of the Linux Documentation Project HTML archive.

[2]All of the pages in these tests announced the following server response header indicating its cacheability:

Cache-Control: max-age=3600

[3]A note on the compressed files: all compression was performed dynamically by mod_gzip for Apache/1.3.27.

How much improvement can you see with compression? The difference in measured download times on a very lightly loaded server indicates that the time to download the Base Page (the initial HTML file) improved by between 1.3 and 1.6 seconds across a very slow connection when compression was used.

Base Page Performance

There is a slightly slower time for the server to respond to a client requesting a compressed page. Measurements show that the median response time for the server averaged 0.23 seconds for the uncompressed page and 0.27 seconds for the compressed page. However, most Web server administrators should be willing to accept a 0.04 increase in response time to achieve a 1.5 second improvement in file transfer time.

First Byte Performance

Web pages are not completely HTML. How do improved HTML (and CSS) download times affect overall performance? The graph below shows that overall download times for the test page were 1 to 1.5 seconds better when the HTML files were compressed.

Total Page Performance

To further emphasize the value of compression, I ran a test on a Web server to see what the average compression ratio would be when requesting a very large number of files. As well, I wanted to determine what the affect on server response time would be when requesting large numbers of compressed files simultaneously. There were 1952 HTML files in the test directory and I checked the results using CURL across my local LAN.[1]

Large sample of File Requests (1952 HTML Files)

mod_gzip

mod_deflate[2]

Average Compression by File Size mod_gzip mod_deflate 0-999 0.713 0.777[3] 1000-4999 0.440 0.440 5000-9999 0.389 0.389 10000-19999 0.369 0.369 20000-49999 0.350 0.350 50000 and up 0.329 0.331

Conclusion

With a few simple commands, and a little bit of configuration, an Apache Web server can be configured to deliver a large amount of content in a compressed format. These benefits are not simply limited to static pages; dynamic pages generated by PHP and other dynamic content generators can be compressed by using the Apache compression modules. When added other performance tuning mechanisms and appropriate server-side caching rules, these modules can substantially reduce the bandwidth for a very low cost.

[1] The files were the top level HTML files from the Linux Documentation Project. They were installed on an Apache 1.3.27 server running mod_gzip and an Apache 2.0.44 server using mod_deflate. Minimum file size was 80 bytes and maximum file size was 99419 bytes.

[2] mod_deflate for Apache/2.0.44 and earlier comes with the compression ratio set for Best Speed, not Best Compression. This configuration can be modified using the tips found here; and starting with Apache/2.0.45, there will be a configuration directive that will allow admins to configure the compression ratio that they want.

In this example, the compression ratio was set to Level 6.

[3] mod_deflate does not have a lower bound for file size, so it attempts to compress files that are too small to benefit from compression. This results in files smaller than approximately 120 bytes becoming larger when processed by mod_deflate.

NOTE: This hack is only relevant to Apache 2.0.44 or lower. Starting with Apache 2.0.45, the server contains the DeflateCompressionLevel directive, which allows for user-configured compression levels in the httpd.conf file.

One of the complaints leveled against mod_deflate for Apache 2.0.44 and below has been the lower compression ratio that it produces when compared to mod_gzip for Apache 1.3.x and 2.0.x. This issue has been traced to a decision made by the author of mod_deflate to focus on fast compression versus maximum compression.

In discussions with the author of mod_deflate and the maintainer of mod_gzip, the location of the issue was quickly found. The level of compression can be easily modified by changing the ZLIB compression setting in mod_deflate.c from Z_BEST_SPEED (equivalent to “zip -1″) to Z_BEST_COMPRESSION (equivalent to “zip -9″). These defaults can also be replaced with a numeric value between 1 and 9. A “hacked” version of the mod_deflate.c code is available here. In this file, the compression level has been set to 6, which is regarded as a good balance between speed and compression (and also happens to be ZLIB’s default ratio). Some other variations are highlighted below.

zRC = deflateInit2(&ctx->stream, Z_BEST_SPEED, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

1. zRC = deflateInit2(&ctx->stream, Z_BEST_COMPRESSION, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

2. zRC = deflateInit2(&ctx->stream, 6, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

3. zRC = deflateInit2(&ctx->stream, 9, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

In a previous paper, the use of mod_gzip to dynamically compress the output from an Apache server. With the growing use of the Apache 2.0.x family of Web servers, the question arises of how to perform a similar GZIP-encoding function within this server. The developers of the Apache 2.0.x servers have included a module in the codebase for the server to perform just this task.

mod_deflate is included in the Apache 2.0.x source package, and compiling it in is a simple matter of adding it to the configure command.

	./configure --enable-modules=all --enable-mods-shared=all --enable-deflate

When the server is made and installed, the GZIP-encoding of documents can be enabled in one of two ways: explicit exclusion of files by extension; or by explcit inclusion of files by MIME type. These methods are specified in the httpd.conf file.

Explicit Exclusion

SetOutputFilter DEFLATE
DeflateFilterNote ratio
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI .(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI .pdf$ no-gzip dont-vary

Explicit Inclusion

DeflateFilterNote ratio
AddOutputFilterByType DEFLATE text/*
AddOutputFilterByType DEFLATE application/ms* application/vnd* application/postscript

If a client sends a request which does not include the Accept-Encoding: gzip header, then the item which is stored in the cache cannot be returned to the requesting client if the Accept-Encoding headers do not match. The request must then be passed directly to the origin server to obtain a non-encoded version. In effect, proxy servers may store 2 or more copies of the same file, depending on the client request conditions which cause the server response to Vary.

Removing the Vary response requirement for objects not handled means that if the objects do not vary due to any other directives on the server (browser type, for example), then the cached object can be served up without any additional requests until the Time-To-Live (TTL) of the cached object has expired.

In examining the performance of mod_deflate against mod_gzip, the one item that distinguished mod_deflate from mod_gzip in versions of Apache prior to 2.0.45 was the amount of compression that occurred. The examples below demonstrate that the compression algorithm for mod_gzip produces between 4-6% more compression than mod_deflate for the same file.[1]

Table 1 — /compress/homepage2.html

Table 2 — /documents/spierzchala-resume.ps

Attempts to increase the compression ratio of mod_deflate in Apache 2.044 and lower using the directives provided for this module produced no further decrease in transferred file size. A comment from one of the authors of the mod_deflate module stated that the module was written specifically to ensure that server performance was not degraded by using this compression method. The module was, by default, performing the fastest compression possible, rather than a mid-range compromise between speed and final file size.

Starting with Apache 2.0.45, the compression level of mod_deflate is configurable using the DeflateCompressionLevel directive. This directive accepts values between 1 (fastest compression speed; lowest compression ratio) and 9 (slowest compression speed; highest compression ratio), with the default value being 6. This simple change makes the compression in mod_deflate comparable to mod_gzip out of the box.

Using mod_deflate for Apache 2.0.x is a quick and effective way to decrease the size of the files that are sent to clients. Anything that can produce between 50% and 80% in bandwidth savings with so little effort should definitely be considered for any and all Apache 2.0.x deployments wishing to use the default Apache codebase.

[1] A note on the compression in mod_deflate for Apache 2.044 and lower: The level of compression can be modified by changing the ZLIB compression setting in mod_deflate.c from Z_BEST_SPEED (equivalent to “gzip -1″) to Z_BEST_COMPRESSION (equivalent to “gzip -9″). These defaults can also be replaced with a numeric value between 1 and 9.

More info on hacking mod_deflate for Apache 2.0.44 and lower can be found here.

A little-used or discussed feature of PHP is the ability to compress output from the scripts using GZIP for more efficient transfer to requesting clients. By automatically detecting the ability of the requesting clients to accept and interpret GZIP encoded HTML, PHP4 can decrease the size of files transferred to the client by 60% to 80%.

The information given here is known to work on systems running Red Hat 8.0, Apache/1.3.27, Apache/2.0.44 and PHP/4.3.1.

[Note: Although not re-tested since this article was originally written, compression is still present in the PHP 5.x releases and can be used to effectively compress content on shared or hosted servers where compression is not enabled within the Web server.]

Configuring PHP

The configuration needed to make this work is simple. Check your installed Red Hat RPMS for the following two packages:

1. zlib
2. zlib-devel

For those not familiar with zlib, it is a highly efficient, open-source compression library. This library is used by PHP uses to compress the output sent to the client.

Compile PHP4 with your favourite ./configure statement. I use the following:

Apache/1.3.27
./configure –without-mysql –with-apxs=/usr/local/apache/bin/apxs –with-zlib

Apache/2.0.44
./configure –without-mysql –with-apxs2=/usr/local/apache2/bin/apxs –with-zlib

After doing make && make install, PHP4 should be ready to go as a dynamic Apache module. Now, you have to make some modifications to the php.ini file. This is usually found in /usr/local/lib, but if it’s not there, don’t panic; you will find some php.ini* files in the directory where you unpacked PHP4. Simply copy one of those to /usr/local/lib and rename it php.ini.

Within php.ini, some modifications need to be made to switch on the GZIP compression detection and encoding. There are two methods to do this.

Method 1:

output_buffering = On
output_handler = ob_gzhandler
zlib.output_compression = Off

Method 2:

output_buffering = Off
output_handler =
zlib.output_compression = On

The ob_gzhandler

The most important component of this entire process is placing the ob_gzhandler PHP command on the page itself. It needs to be placed in the code at the top of the page, above the HTML tag in order to work. It takes the addition of the following line to complete the process:

<?php ob_start("ob_gzhandler"); ?>

In Wordpress installs, this becomes the first line in the HEADER.PHP file. But be careful to check that it’s working properly. If the Web application has the compression function built into it, and you add the ob_gzhandler function, a funky error message will appear at the top of the page telling you that your can’t invoke compression twice.

Web servers with native compression are smarter than that – they realize that the file is already compressed and don’t run it through the compression algorithm again.

Once this is in place, you will be able to verify the decrease in size using any HTTP browser capture tool (Firebug, Safari Web Inspector, Fiddler2, etc.)

So?

The winning situation here is that for an expenditure of $0 (except your time) and a tiny bit more server overhead (you’re probably still using fewer resources than if you were running ASP on IIS!), you will now be sending much smaller, dynamically generated html documents to your clients, reducing your bandwidth usage and the amount of time it takes to download the files.

How much of a size reduction is achieved? Well, I ran a test on my Web server, using WGET to retrieve the file. The configuration and results of the test are listed below.

Method 0: No Compression wget www.pierzchala.com/resume.php	File Size: 9415 bytes
Method 1: ob_gzhandler wget –header=”Accept-Encoding: gzip,*” www.pierzchala.com/resume.php	File Size: 3529 bytes
Method 2: zlib.output_compression wget –header=”Accept-Encoding: gzip,*” www.pierzchala.com/resume.php	File Size: 3584 bytes

You will have to experiment with the method that give the most efficient balance between file size and overhead and processing time on your server.

A 62% reduction in transferred file size without affecting the quality of the data sent to the client is a pretty good return for 10 minutes of work. I recommend including this procedure in all of your future PHP4 builds.

I have been running the GrabPERF Compression and Performance study for less than a week, but I thought that I should share some of the initial results with everyone.

As you can see above, the byte transmission savings gained by some sites is pretty astounding. Google News sends a pages with a median weight of near 31,000 bytes when compressed; but when compression is disabled on the client, this jumps to over 139,000 bytes.

What is interesting is that the performance gains don’t look truly significant. However, they compressed pages are faster, and have the added benefit of costing the site less, as bandwidth costs count by the byte (I know it’s more complicated than that, but for now, let’s assume a fantasy world).

I will continue to monitor that results and will close the measurements after 14 days and write up a final report.

Technorati Tags: GrabPERF, Web+performance, Web+Compression, HTTP+Compression, Performance+tuning, GZIP