It’s always funny when somewhat tech-savvy folks purposely make their bandwidth bills higher than they need to be.

Here’s TechCrunch’s HTTP header response.

HTTP/1.1 200 OK
Date: Tue, 16 Jan 2007 16:02:23 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 PHP/5.2.0-8 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8
X-Pingback: http://www.techcrunch.com/xmlrpc.php
Status: 200 OK
Transfer-Encoding: chunked
Content-Type: text/html; charset="UTF-8"

Port80 Software’s Compression Checker gives us some idea how much bandwidth Mr. Arrington, et al. could save just by activating this little feature, which comes baked into Apache 2.2.x.

Turn. On. Mod_deflate.

How much improvement can you see with compression? The difference in measured download times on a very lightly loaded server indicates that the time to download the Base Page (the initial HTML file) improved by between 1.3 and 1.6 seconds across a very slow connection when compression was used.

Base Page Performance

There is a slightly slower time for the server to respond to a client requesting a compressed page. Measurements show that the median response time for the server averaged 0.23 seconds for the uncompressed page and 0.27 seconds for the compressed page. However, most Web server administrators should be willing to accept a 0.04 increase in response time to achieve a 1.5 second improvement in file transfer time.

First Byte Performance

Web pages are not completely HTML. How do improved HTML (and CSS) download times affect overall performance? The graph below shows that overall download times for the test page were 1 to 1.5 seconds better when the HTML files were compressed.

Total Page Performance

To further emphasize the value of compression, I ran a test on a Web server to see what the average compression ratio would be when requesting a very large number of files. As well, I wanted to determine what the affect on server response time would be when requesting large numbers of compressed files simultaneously. There were 1952 HTML files in the test directory and I checked the results using CURL across my local LAN.[1]

Large sample of File Requests (1952 HTML Files)

mod_gzip

mod_deflate[2]

Average Compression by File Size mod_gzip mod_deflate 0-999 0.713 0.777[3] 1000-4999 0.440 0.440 5000-9999 0.389 0.389 10000-19999 0.369 0.369 20000-49999 0.350 0.350 50000 and up 0.329 0.331

Conclusion

With a few simple commands, and a little bit of configuration, an Apache Web server can be configured to deliver a large amount of content in a compressed format. These benefits are not simply limited to static pages; dynamic pages generated by PHP and other dynamic content generators can be compressed by using the Apache compression modules. When added other performance tuning mechanisms and appropriate server-side caching rules, these modules can substantially reduce the bandwidth for a very low cost.

[1] The files were the top level HTML files from the Linux Documentation Project. They were installed on an Apache 1.3.27 server running mod_gzip and an Apache 2.0.44 server using mod_deflate. Minimum file size was 80 bytes and maximum file size was 99419 bytes.

[2] mod_deflate for Apache/2.0.44 and earlier comes with the compression ratio set for Best Speed, not Best Compression. This configuration can be modified using the tips found here; and starting with Apache/2.0.45, there will be a configuration directive that will allow admins to configure the compression ratio that they want.

In this example, the compression ratio was set to Level 6.

[3] mod_deflate does not have a lower bound for file size, so it attempts to compress files that are too small to benefit from compression. This results in files smaller than approximately 120 bytes becoming larger when processed by mod_deflate.

NOTE: This hack is only relevant to Apache 2.0.44 or lower. Starting with Apache 2.0.45, the server contains the DeflateCompressionLevel directive, which allows for user-configured compression levels in the httpd.conf file.

One of the complaints leveled against mod_deflate for Apache 2.0.44 and below has been the lower compression ratio that it produces when compared to mod_gzip for Apache 1.3.x and 2.0.x. This issue has been traced to a decision made by the author of mod_deflate to focus on fast compression versus maximum compression.

In discussions with the author of mod_deflate and the maintainer of mod_gzip, the location of the issue was quickly found. The level of compression can be easily modified by changing the ZLIB compression setting in mod_deflate.c from Z_BEST_SPEED (equivalent to “zip -1″) to Z_BEST_COMPRESSION (equivalent to “zip -9″). These defaults can also be replaced with a numeric value between 1 and 9. A “hacked” version of the mod_deflate.c code is available here. In this file, the compression level has been set to 6, which is regarded as a good balance between speed and compression (and also happens to be ZLIB’s default ratio). Some other variations are highlighted below.

zRC = deflateInit2(&ctx->stream, Z_BEST_SPEED, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

1. zRC = deflateInit2(&ctx->stream, Z_BEST_COMPRESSION, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

2. zRC = deflateInit2(&ctx->stream, 6, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

3. zRC = deflateInit2(&ctx->stream, 9, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

In a previous paper, the use of mod_gzip to dynamically compress the output from an Apache server. With the growing use of the Apache 2.0.x family of Web servers, the question arises of how to perform a similar GZIP-encoding function within this server. The developers of the Apache 2.0.x servers have included a module in the codebase for the server to perform just this task.

mod_deflate is included in the Apache 2.0.x source package, and compiling it in is a simple matter of adding it to the configure command.

	./configure --enable-modules=all --enable-mods-shared=all --enable-deflate

When the server is made and installed, the GZIP-encoding of documents can be enabled in one of two ways: explicit exclusion of files by extension; or by explcit inclusion of files by MIME type. These methods are specified in the httpd.conf file.

Explicit Exclusion

SetOutputFilter DEFLATE
DeflateFilterNote ratio
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI .(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI .pdf$ no-gzip dont-vary

Explicit Inclusion

DeflateFilterNote ratio
AddOutputFilterByType DEFLATE text/*
AddOutputFilterByType DEFLATE application/ms* application/vnd* application/postscript

If a client sends a request which does not include the Accept-Encoding: gzip header, then the item which is stored in the cache cannot be returned to the requesting client if the Accept-Encoding headers do not match. The request must then be passed directly to the origin server to obtain a non-encoded version. In effect, proxy servers may store 2 or more copies of the same file, depending on the client request conditions which cause the server response to Vary.

Removing the Vary response requirement for objects not handled means that if the objects do not vary due to any other directives on the server (browser type, for example), then the cached object can be served up without any additional requests until the Time-To-Live (TTL) of the cached object has expired.

In examining the performance of mod_deflate against mod_gzip, the one item that distinguished mod_deflate from mod_gzip in versions of Apache prior to 2.0.45 was the amount of compression that occurred. The examples below demonstrate that the compression algorithm for mod_gzip produces between 4-6% more compression than mod_deflate for the same file.[1]

Table 1 — /compress/homepage2.html

Table 2 — /documents/spierzchala-resume.ps

Attempts to increase the compression ratio of mod_deflate in Apache 2.044 and lower using the directives provided for this module produced no further decrease in transferred file size. A comment from one of the authors of the mod_deflate module stated that the module was written specifically to ensure that server performance was not degraded by using this compression method. The module was, by default, performing the fastest compression possible, rather than a mid-range compromise between speed and final file size.

Starting with Apache 2.0.45, the compression level of mod_deflate is configurable using the DeflateCompressionLevel directive. This directive accepts values between 1 (fastest compression speed; lowest compression ratio) and 9 (slowest compression speed; highest compression ratio), with the default value being 6. This simple change makes the compression in mod_deflate comparable to mod_gzip out of the box.

Using mod_deflate for Apache 2.0.x is a quick and effective way to decrease the size of the files that are sent to clients. Anything that can produce between 50% and 80% in bandwidth savings with so little effort should definitely be considered for any and all Apache 2.0.x deployments wishing to use the default Apache codebase.

[1] A note on the compression in mod_deflate for Apache 2.044 and lower: The level of compression can be modified by changing the ZLIB compression setting in mod_deflate.c from Z_BEST_SPEED (equivalent to “gzip -1″) to Z_BEST_COMPRESSION (equivalent to “gzip -9″). These defaults can also be replaced with a numeric value between 1 and 9.

More info on hacking mod_deflate for Apache 2.0.44 and lower can be found here.

I just had a look at the list of domains that I own and realize that
I want/need to clean house. There are 3 domains that I want to keep,
and 9 that I am looking to get rid of.

Domains that I want to unload include:

• mod-deflate.net
• mod-deflate.org
• performancecore.org
• performancecorps.org
• performancefreaks.org
• performanceguru.org
• performanceindex.org
• searchindex.org
• webcaching.org
• webcompression.org
• webperformance.org

Currently asking $2,000 each for webperformance.org, webcaching.org, searchindex.org, performanceindex.org, and webcompression.org, and anything I can get for the others.

edgeio-key: 00878878ccc0ba01e6db77a82889f12e12fcb084

Technorati Tags: listing, Web performance, domains for sale, for sale, domain name

Opera, my #2 browser choice is now free. Go get it!

Now, Netcraft reports that the site is slowing down as a result. I am also measuring it using GrabPERF

—- START RANT

Now, if they turned on compression and had some explicit cache-control information, they might not be doing to badly.

HTTP/1.1 200 OK
Date: Thu, 22 Sep 2005 20:51:09 GMT
Server: mod-xslt/1.3.8 Apache/2.0.54 (Debian GNU/Linux) \
                           mod_perl/1.999.21 Perl/v5.8.4
Last-Modified: Thu, 22 Sep 2005 12:25:37 GMT
ETag: "771846-1277-63ed7240"
Accept-Ranges: bytes
Content-Length: 4727
Content-Type: text/html

They are using Apache. Apache has mod_deflate. TURN IT ON!

—- END RANT

Ummm, this one floored me. If Weblogs.com wants to reduce their traffic, turn on mod_deflate in Apache 2.0.

HTTP/1.1 200 OK
Date: Fri, 02 Sep 2005 14:13:10 GMT
Server: Apache/2.0.46 (Red Hat)
Last-Modified: Fri, 02 Sep 2005 14:09:54 GMT
ETag: "ac0b3-3559d-840c3080"
Accept-Ranges: bytes
Content-Length: 218525
Connection: close
Content-Type: text/html; charset=UTF-8

I know it won’t solve everything, but it’s a start.

Ok, one of the tests that the GrabPERF System is running is doing a simple search on the Technorati site.

Ouch.

Now, as I mentioned before, Technorati has some interesting things going on in their www servers. For the Web geeks out there, here is what their headers look like.

HTTP/1.0 200 OK
Date: Mon, 11 Jul 2005 16:15:47 GMT
Server: Apache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
X-Powered-By: PHP/4.3.11
Set-Cookie: TECHNORATI_MEMBER=deleted; \
                   expires=Sun, 11 Jul 2004 16:15:46 GMT; path=/
Set-Cookie: TECHNORATI_MEMBER=[blah-blah]; \
                   expires=Wed, 12 Oct 2005 16:15:47 GMT; path=/; \
                   domain=technorati.com
Pragma: no-cache
Content-Type: text/html; charset=UTF-8
Connection: close

The Cache-Control header I can understand. The Pragma: no-cache is basically useless, as the Pragma header is only considered as a valid client-side header.

But the use of HTTP/1.0 and the explicit Connection: close really bother me. Removing the ability for clients to maintain persistent connections is a Network and Server resource issue, and should be avoided at all costs.

And where’s the compression? There is no need to send raw content to clients that understand and can process compressing text content. Using either hardware or mod_gzip or mod_deflate, the size of transferred content can be very easily reduced.

Sorry for the rant, but I seem to be on a Web performance kick lately. And the team at Technorati is one that I would expect to have gotten the need for this Web performance thing.

Technorati: web performance, HTTP, Technorati

Here is a little tidbit that we discovered while trying to debug an issue at work. One of my colleagues found that the Symantec/Norton Personal Firewall/Internet Security mangles the “Accept-Encoding” header sent out by any application — browser, streaming media, etc.

More can be found here.

This is a serious problem, and has a negative effect on Web performance in general, as one of the key methods for improving bandwidth consumption and user performance is Server-Side Compression of as much content as possible.

What the client wants to send:  Accept-Encoding: gzip,deflate\r\n
What is sent:                   ---------------: ---- -------\r\n

What is the problem? Is this because Symantec can’t parse compressed content on the fly?

For those of you ourt there that use SiteMeter, I need some verification that the service is unreachable right now. The icon does not load on my Blog page (you may have noticed this if you are reading this is a browser) and when you do get to the site, the stats database is unreachable.

Wouldn’t bother me much, except I am a paying subscriber of this service. And companies who offer for-pay services to this community should read Mary Hodder’s rant on lost data to understand that a lot of people see this information as a NEED to have.

Seems that they have configured their server with a broken set of re-direction rules that sens Firefox into an endless loop of HTTP 302 messages to the same page.

http://www.sitemeter.com/default.asp?action=LOGINFORM

GET /default.asp?action=LOGINFORM HTTP/1.1Host: www.sitemeter.comUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5Accept-Language: en-us,en;q=0.5Accept-Encoding: gzip,deflateAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7Keep-Alive: 300Connection: keep-aliveCookie: USERID2=XXXXXXXXXXX; PASSWORD2=XXXXXXX; ASPSESSIONIDSSCDARAD=GLJMJECBGJPGPAFDLLIINGAM

HTTP/1.x 302 Object movedCache-Control: privateConnection: closeDate: Fri, 17 Dec 2004 13:22:04 GMTContent-Length: 149Content-Type: text/htmlLocation: default.asp?action=LOGINFORMServer: Microsoft-IIS/6.0X-Powered-By: ASP.NETSet-Cookie: USERID=; expires=Wednesday, 09-Nov-99 23:12:40 GMT; domain=www.sitemeter.com;Set-Cookie: PASSWORD=; expires=Wednesday, 09-Nov-99 23:12:40 GMT; domain=www.sitemeter.com;Set-Cookie: PASSWORD2=; expires=Mon, 01-Jan-2001 05:00:00 GMT; path=/Set-Cookie: USERID2=; expires=Mon, 01-Jan-2001 05:00:00 GMT; path=/

MSIE Lets you get to the homepage and then announces that the database for my site is unavailable.

ADDENDUM (JAN 8, 2005): Why is this article suddenly getting so many hits? Today, this single article has leapt to the lead of the greatest hits chart. Wonder what’s up….