This paper is an extension of the work done for another article that highlighted the performance benefits of retrieving uncompressed and compressed objects directly from the origin server. I wanted to add a proxy server into the stream and determine if proxy servers helped improve the performance of object downloads, and by how much.

Using the same series of objects in the original compression article[1], the CURL tests were re-run 3 times:

 

1. Directly from the origin server
2. Through the proxy server, to load the files into cache
3. Through the proxy server, to avoid retrieving files from the origin.[2]

This series of three tests was repeated twice: once for the uncompressed files, and then for the compressed objects.[3]

As can be seen clearly in the plots below, compression caused web page download times to improve greatly, when the objects were retrieved from the source. However, the performance difference between compressed and uncompressed data all but disappears when retrieving objects from a proxy server on a corporate LAN.

Instead of the linear growth between object size and download time seen in both of the retrieval tests that used the origin server (Source and Proxy Load data), the Proxy Draw data clearly shows the benefits that accrue when a proxy server is added to a network to assist with serving HTTP traffic.

	MEAN DOWNLOAD TIME
Uncompressed Pages
Total Time Uncompressed — No Proxy	0.256
Total Time Uncompressed — Proxy Load	0.254
Total Time Uncompressed — Proxy Draw	0.110
Compressed Pages
Total Time Compressed — No Proxy	0.181
Total Time Compressed — Proxy Load	0.140
Total Time Compressed — Proxy Draw	0.104

The data above shows just how much of an improvement is gained by adding a local proxy server, explicit caching descriptions and compression can add to a Web site. For sites that do force a great of requests to be returned directly to the origin server, compression will be of great help in reducing bandwidth costs and improving performance. However, by allowing pages to be cached in local proxy servers, the difference between compressed and uncompressed pages vanishes.

Conclusion

Compression is a very good start when attempting to optimize performance. The addition of explicit caching messages in server responses which allow proxy servers to serve cached data to clients on remote local LANs can improve performance to even a greater extent than compression can. These two should be used together to improve the overall performance of Web sites.

---

[1]The test set was made up of the 1952 HTML files located in the top directory of the Linux Documentation Project HTML archive.

[2]All of the pages in these tests announced the following server response header indicating its cacheability:

Cache-Control: max-age=3600

[3]A note on the compressed files: all compression was performed dynamically by mod_gzip for Apache/1.3.27.

Tags: ads, apache, app, ARIN, bandwidth, cache-control, Cacheability, caching, client, compress, compression, control, corporate, data, EAD, extension, gzip, HTML, HTTP, IE, IM, improvement, IP, it, LAN, Linux, mod_gzip, Om, One, optimize, origin server, Pages, performance, performance improvement, pr, proxy, proxy server, run, server, the Origin, The origin server, traffic, web, wordpress

The last time I attempted to compile mod_gzip into Apache, I found that the instructions for doing so were not documented clearly on the project page. After a couple of failed attempts, I finally found the instructions buried at the end of the ChangeLog document.

I present the instructions here to preserve your sanity.

Before you can actually get mod_gzip to work, you have to uncomment it in the httpd.conf file module list (Apache 1.3.x) or add it to the module list (Apache 2.0.x).

---

Now there are two ways to build mod_gzip: statically compiled into Apache and a DSO-File for mod_so. If you want to compile it statically into Apache, just copy the source to Apache src/modules directory and there into a subdirectory named ‘gzip’. You can activate it via a parameter of the configure script.

 ./configure --activate-module=src/modules/gzip/mod_gzip.a
 make
 make install

This will build a new Apache with mod_gzip statically built in.

The DSO-Version is much easier to build.

 make APXS=/path/to/apxs
 make install APXS=/path/to/apxs
 /path/to/apachectl graceful

The apxs script is normally located inside the bin directory of Apache.

Tags: apache, gzip, HTTP, IE, IM, IP, it, mod_gzip, normal, Om, pr

NOTE: This hack is only relevant to Apache 2.0.44 or lower. Starting with Apache 2.0.45, the server contains the DeflateCompressionLevel directive, which allows for user-configured compression levels in the httpd.conf file.

One of the complaints leveled against mod_deflate for Apache 2.0.44 and below has been the lower compression ratio that it produces when compared to mod_gzip for Apache 1.3.x and 2.0.x. This issue has been traced to a decision made by the author of mod_deflate to focus on fast compression versus maximum compression.

In discussions with the author of mod_deflate and the maintainer of mod_gzip, the location of the issue was quickly found. The level of compression can be easily modified by changing the ZLIB compression setting in mod_deflate.c from Z_BEST_SPEED (equivalent to “zip -1″) to Z_BEST_COMPRESSION (equivalent to “zip -9″). These defaults can also be replaced with a numeric value between 1 and 9. A “hacked” version of the mod_deflate.c code is available here. In this file, the compression level has been set to 6, which is regarded as a good balance between speed and compression (and also happens to be ZLIB’s default ratio). Some other variations are highlighted below.

---

Original Code

zRC = deflateInit2(&ctx->stream, Z_BEST_SPEED, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

---

Hacked Code

1. zRC = deflateInit2(&ctx->stream, Z_BEST_COMPRESSION, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

2. zRC = deflateInit2(&ctx->stream, 6, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

3. zRC = deflateInit2(&ctx->stream, 9, Z_DEFLATED, c->windowSize, c->memlevel, Z_DEFAULT_STRATEGY);

---

A change has been made to mod_deflate in Apache 2.0.45 that adds a directive named DeflateCompressionLevel to the mod_deflate options. This will accept a numeric value between 1 (Best Speed) and 9 (Best Compression), with the default set at 6.

Tags: apache, compression, deflate, gzip, mod_gzip

In a previous paper, the use of mod_gzip to dynamically compress the output from an Apache server. With the growing use of the Apache 2.0.x family of Web servers, the question arises of how to perform a similar GZIP-encoding function within this server. The developers of the Apache 2.0.x servers have included a module in the codebase for the server to perform just this task.

mod_deflate is included in the Apache 2.0.x source package, and compiling it in is a simple matter of adding it to the configure command.

	./configure --enable-modules=all --enable-mods-shared=all --enable-deflate

When the server is made and installed, the GZIP-encoding of documents can be enabled in one of two ways: explicit exclusion of files by extension; or by explcit inclusion of files by MIME type. These methods are specified in the httpd.conf file.

---

Explicit Exclusion

SetOutputFilter DEFLATE
DeflateFilterNote ratio
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI .(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI .pdf$ no-gzip dont-vary

---

Explicit Inclusion

DeflateFilterNote ratio
AddOutputFilterByType DEFLATE text/*
AddOutputFilterByType DEFLATE application/ms* application/vnd* application/postscript

---

Both methods enable the automatic GZIP-encoding of all MIME-types, except image and PDF files, as they leave the server. Image files and PDF files are excluded as they are already in a highly compressed format. In fact, PDFs become unreadable by Adobe’s Acrobat Reader if they are further compressed by mod_deflate or mod_gzip.On the server used for testing mod_deflate for this article, no Windows executables or compressed files are served to visitors. However, for safety’s sake, please ensure that compressed files and binaries are not GZIP-encoded by your Web server application.For the file-types indicated in the exclude statements, the server is told explicitly not to send the Vary header. The Vary header indicates to any proxy or cache server which particular condition(s) will cause this response to Vary from other responses to the same request.

If a client sends a request which does not include the Accept-Encoding: gzip header, then the item which is stored in the cache cannot be returned to the requesting client if the Accept-Encoding headers do not match. The request must then be passed directly to the origin server to obtain a non-encoded version. In effect, proxy servers may store 2 or more copies of the same file, depending on the client request conditions which cause the server response to Vary.

Removing the Vary response requirement for objects not handled means that if the objects do not vary due to any other directives on the server (browser type, for example), then the cached object can be served up without any additional requests until the Time-To-Live (TTL) of the cached object has expired.

In examining the performance of mod_deflate against mod_gzip, the one item that distinguished mod_deflate from mod_gzip in versions of Apache prior to 2.0.45 was the amount of compression that occurred. The examples below demonstrate that the compression algorithm for mod_gzip produces between 4-6% more compression than mod_deflate for the same file.[1]

Table 1 — /compress/homepage2.html

Compression	Size	Compression %
No compression	56380 bytes	n/a
Apache 1.3.x/mod_gzip	16333 bytes	29% of original
Apache 2.0.x/mod_deflate	19898 bytes	35% of original

Table 2 — /documents/spierzchala-resume.ps

Compression	Size	Compression %
No Compression	63451 bytes	n/a
Apache 1.3.x/mod_gzip	19758 bytes	31% of original
Apache 2.0.x/mod_deflate	23407 bytes	37% of original

Attempts to increase the compression ratio of mod_deflate in Apache 2.044 and lower using the directives provided for this module produced no further decrease in transferred file size. A comment from one of the authors of the mod_deflate module stated that the module was written specifically to ensure that server performance was not degraded by using this compression method. The module was, by default, performing the fastest compression possible, rather than a mid-range compromise between speed and final file size.

Starting with Apache 2.0.45, the compression level of mod_deflate is configurable using the DeflateCompressionLevel directive. This directive accepts values between 1 (fastest compression speed; lowest compression ratio) and 9 (slowest compression speed; highest compression ratio), with the default value being 6. This simple change makes the compression in mod_deflate comparable to mod_gzip out of the box.

Using mod_deflate for Apache 2.0.x is a quick and effective way to decrease the size of the files that are sent to clients. Anything that can produce between 50% and 80% in bandwidth savings with so little effort should definitely be considered for any and all Apache 2.0.x deployments wishing to use the default Apache codebase.

---

[1] A note on the compression in mod_deflate for Apache 2.044 and lower: The level of compression can be modified by changing the ZLIB compression setting in mod_deflate.c from Z_BEST_SPEED (equivalent to “gzip -1″) to Z_BEST_COMPRESSION (equivalent to “gzip -9″). These defaults can also be replaced with a numeric value between 1 and 9.

More info on hacking mod_deflate for Apache 2.0.44 and lower can be found here.

Tags: Acrobat, Adobe, apache, Apache server, cache server, compression, compression algorithm, compression solutions, deflate, gif, gzip, HTML, mod_gzip, origin server, Web page compression, Web server, Web servers, Windows

Ok, one of the tests that the GrabPERF System is running is doing a simple search on the Technorati site.

Ouch.

Now, as I mentioned before, Technorati has some interesting things going on in their www servers. For the Web geeks out there, here is what their headers look like.

HTTP/1.0 200 OK
Date: Mon, 11 Jul 2005 16:15:47 GMT
Server: Apache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
X-Powered-By: PHP/4.3.11
Set-Cookie: TECHNORATI_MEMBER=deleted; \
                   expires=Sun, 11 Jul 2004 16:15:46 GMT; path=/
Set-Cookie: TECHNORATI_MEMBER=[blah-blah]; \
                   expires=Wed, 12 Oct 2005 16:15:47 GMT; path=/; \
                   domain=technorati.com
Pragma: no-cache
Content-Type: text/html; charset=UTF-8
Connection: close

The Cache-Control header I can understand. The Pragma: no-cache is basically useless, as the Pragma header is only considered as a valid client-side header.

But the use of HTTP/1.0 and the explicit Connection: close really bother me. Removing the ability for clients to maintain persistent connections is a Network and Server resource issue, and should be avoided at all costs.

And where’s the compression? There is no need to send raw content to clients that understand and can process compressing text content. Using either hardware or mod_gzip or mod_deflate, the size of transferred content can be very easily reduced.

Sorry for the rant, but I seem to be on a Web performance kick lately. And the team at Technorati is one that I would expect to have gotten the need for this Web performance thing.

---

Technorati: web performance, HTTP, Technorati

Tags: apache, cache-control, client, client-side, compress, compressing, compression, content, control, deflate, EAD, GrabPERF, gzip, HTML, HTTP, IE, IM, IP, it, LAN, maintain, media, mod_gzip, Om, One, performance, persistent, persistent connections, PHP, pr, run, search, server, Technorati, web, Web Performance, Web+performance

This paper is an extension of the work done for another article that highlighted the performance benefits of retrieving uncompressed and compressed objects directly from the origin server. I wanted to add a proxy server into the stream and determine if proxy servers helped improve the performance of object downloads, and by how much.

Using the same series of objects in the original compression article^[1], the CURL tests were re-run 3 times:

1. Directly from the origin server
2. Through the proxy server, to load the files into cache
3. Through the proxy server, to avoid retrieving files from the origin.^[2]

eries of three tests was repeated twice: once for the uncompressed files, and then for the compressed objects.^[3]

As can be seen clearly in the plots below, compression caused web page download times to improve greatly, when the objects were retrieved from the source. However, the performance difference between compressed and uncompressed data all but disappears when retrieving objects from a proxy server on a corporate LAN.

Instead of the linear growth between object size and download time seen in both of the retrieval tests that used the origin server (Source and Proxy Load data), the Proxy Draw data clearly shows the benefits that accrue when a proxy server is added to a network to assist with serving HTTP traffic.

Uncompressed Pages

	MEAN DOWNLOAD TIME
Total Time Uncompressed — No Proxy	0.256
Total Time Uncompressed — Proxy Load	0.254
Total Time Uncompressed — Proxy Draw	0.110

Compressed Pages

Total Time Compressed — No Proxy	0.181
Total Time Compressed — Proxy Load	0.140
Total Time Compressed — Proxy Draw	0.104

The data above shows just how much of an improvement is gained by adding a local proxy server, explicit caching descriptions and compression can add to a Web site. For sites that do force a great of requests to be returned directly to the origin server, compression will be of great help in reducing bandwidth costs and improving performance. However, by allowing pages to be cached in local proxy servers, the difference between compressed and uncompressed pages vanishes.

Conclusion

Compression is a very good start when attempting to optimize performance. The addition of explicit caching messages in server responses which allow proxy servers to serve cached data to clients on remote local LANs can improve performance to even a greater extent than compression can. These two should be used together to improve the overall performance of Web sites.

---

[1] The test set was made up of the 1952 HTML files located in the top directory of the Linux Documentation Project HTML archive.

[2] All of the pages in these tests announced the following server response header indicating its cacheability: Cache-Control: max-age=3600

[3] A note on the compressed files: all compression was performed dynamically by mod_gzip for Apache/1.3.27.

Tags: ads, apache, app, bandwidth, cache-control, Cacheability, caching, client, compress, compression, control, corporate, data, EAD, extension, gzip, HTML, HTTP, IE, IM, improvement, IP, it, LAN, Linux, media, mod_gzip, Om, One, optimize, origin server, Pages, performance, performance improvement, PHP, pr, proxy, proxy server, run, server, the Origin, The origin server, traffic, web

The last time I attempted to compile mod_gzip into Apache, I found that the instructions for doing so were not documented clearly on the project page. After a couple of failed attempts, I finally found the instructions buried at the end of the ChangeLog document.

I present the instructions here to preserve your sanity.

Before you can actually get mod_gzip to work, you have to uncomment it in the httpd.conf file module list (Apache 1.3.x) or add it to the module list (Apache 2.0.x).

---

Now there are two ways to build mod_gzip: statically compiled into Apache and a DSO-File for mod_so. If you want to compile it statically into Apache, just copy the source to Apache src/modules directory and there into a subdirectory named ‘gzip’. You can activate it via a parameter of the configure script.

	./configure --activate-module=src/modules/gzip/mod_gzip.a
	make
	make install
	

This will build a new Apache with mod_gzip statically built in.

The DSO-Version is much easier to build.

	make APXS=/path/to/apxs
	make install APXS=/path/to/apxs
	/path/to/apachectl graceful
	

The apxs script is normally located inside the bin directory of Apache.

Tags: apache, gzip, HTTP, IE, IM, IP, it, mod_gzip, normal, Om, pr

Web page compression is not a new technology, but it has just recently gained higher recognition in the minds of IT administrators and managers because of the rapid ROI it generates. Compression extensions exist for most of the major Web server platforms, but in this article I will focus on the Apache and mod_gzip solution.

The idea behind GZIP-encoding documents is very straightforward. Take a file that is to be transmitted to a Web client, and send a compressed version of the data, rather than the raw file as it exists on the filesystem. Depending on the size of the file, the compressed version can run anywhere from 50% to 20% of the original file size.

In Apache, this can be achieved using a couple of different methods. Content Negotiation, which requires that two separate sets of HTML files be generated — one for clients that can handle GZIP-encoding, and one for those who can’t — is one method. The problem with this solution should be readily apparent: there is no provision in this methodology for GZIP-encoding dynamically-generated pages.

The more graceful solution for administrators who want to add GZIP-encoding to Apache is the use of mod_gzip. I consider it one of the overlooked gems for designing a high-performance Web server. Using this module, configured file types — based on file extension or MIME type — will be compressed using GZIP-encoding after they have been processed by all of Apache’s other modules, and before they are sent to the client. The compressed data that is generated reduces the number of bytes transferred to the client, without any loss in the structure or content of the original, uncompressed document.

mod_gzip can be compiled into Apache as either a static or dynamic module; I have chosen to compile it as a dynamic module in my own server (more compile instructions here). The advantage of using mod_gzip is that this method requires that nothing be done on the client side to make it work. All current browsers — Mozilla, Opera, and even Internet Explorer — understand and can process GZIP-encoded text content.

On the server side, all the server or site administrator has to do is compile the module, edit the appropriate configuration directives that were added to the httpd.conf file, enable the module in the httpd.conf file, and restart the server. In less than 10 minutes, you can be serving static and dynamic content using GZIP-encoding without the need to maintain multiple codebases for clients that can or cannot accept GZIP-encoded documents.

When a request is received from a client, Apache determines if mod_gzip should be invoked by noting if the “Accept-Encoding: gzip” HTTP request header has been sent by the client. If the client sends the header, mod_gzip will automatically compress the output of all configured file types when sending them to the client.

This client header announces to Apache that the client will understand files that have been GZIP-encoded. mod_gzip then processes the outgoing content and includes the following server response headers.

		Content-Type: text/html
		Content-Encoding: gzip
		

These server response headers announce that the content returned from the server is GZIP-encoded, but that when the content is expanded by the client application, it should be treated as a standard HTML file. Not only is this successful for static HTML files, but this can be applied to pages that contain dynamic elements, such as those produced by Server-Side Includes (SSI), PHP, and other dynamic page generation methods. You can also use it to compress your Cascading Stylesheets (CSS) and plain text files. As well, a whole range of application file types can be compressed and sent to clients. My httpd.conf file sets the following configuration for the file types handled by mod_gzip:

		mod_gzip_item_include mime ^text/.*
		mod_gzip_item_include mime ^application/postscript$
		mod_gzip_item_include mime ^application/ms.*$
		mod_gzip_item_include mime ^application/vnd.*$
		mod_gzip_item_exclude mime ^application/x-javascript$
		mod_gzip_item_exclude mime ^image/.*$
		

This allows Microsoft Office and Postscript files to be GZIP-encoded, while not affecting PDF files. PDF files should not be GZIP-encoded, as they are already compressed in their native format, and compressing them leads to issues when attempting to display the files in Adobe Acrobat Reader.^[1] For the paranoid system administrator, you may want to explicitly exclude PDF files.

		mod_gzip_item_exclude mime ^application/pdf$
		

Another side-note is that nothing needs to be done to allow the GZIP-encoding of OpenOffice (and presumably, StarOffice) documents. Their MIME-type is already set to text-plain, allowing them to be covered by one of the default rules.

How beneficial is sending GZIP-encoded content? In some simple tests I ran on my Web server using WGET, GZIP-encoded documents showed that even on a small Web server, there is the potential to produce a substantial savings in bandwidth usage.

http://www.pierzchala.com/bio.html	Uncompressed File Size: 3122 bytes
http://www.pierzchala.com/bio.html	Compressed File Size: 1578 bytes
http://www.pierzchala.com/compress/homepage2.html	Uncompressed File Size: 56279 bytes
http://www.pierzchala.com/compress/homepage2.html	Compressed File Size: 16286 bytes

Server administrators may be concerned that mod_gzip will place a heavy burden on their systems as files are compressed on the fly. I argue against that, pointing out that this does not seem to concern the administrators of Slashdot, one of the busiest Web servers on the Internet, who use mod_gzip in their very high-traffic environment.

The mod_gzip project page for Apache 1.3.x is located at SourceForge. The Apache 2.0.x version is available from here.

---

[1] From http://www.15seconds.com/issue/020314.htm

Tags: Acrobat, Adobe, ads, apache, API, app, application, bandwidth, browser, busiest Web servers, client, compress, compressing, compression, content, cro, css, current, data, design, EAD, even Internet Explorer, extension, graceful solution, gzip, HTML, HTTP, IE, IM, Internet, Internet Explorer, IP, ISP, Issues, it, Javascript, maintain, Microsoft, mod_gzip, Om, One, Pages, performance, PHP, posts, pr, run, server, the Origin, traffic, web, Web client, Web page compression, Web server, Web server platforms, Web server using WGET, Web servers, WGET