DNS hijacking is an occurrence that sends fear into the hearts of man and beast. It takes a perfectly harmless (yet critical) process and turns it into a weapon for chaos and mayhem.

This tool, however, does not simply reside in the hands of people looking to maliciously redirect traffic for purposes I can’t quite fathom – I’ll admit, there is still some simple naivete in my Canadian mind.

Legitimate companies, ISPs, service providers also have this tool at their disposal for their own purposes. An useful and accepted version of this exists already in the form of content delivery networks (CDNs) and other third-parties who take a portion of a companies domain name space and use it to deliver distributed edge content and computing, web analytics, or advertising services.

But let’s move this inside the firewall or into the consumer ISP space. These companies provide DNS for millions of customers. As a result, they could easily re-write DNS entries to reflect their own needs rather than those of the consumer.

In the case of corporate IT networks, there is not much that can be done – they own the wire, hardware and software being used, so they will claim it’s part of the corporate IT policy that all employees sign and that will be that.

Consumers, on the other hand, should expect free and unencumbered access to the worldwide DNS network, without being intercepted or redirected by their own ISPs. And there is frankly no way to verify that this is not happening unless you run your own caching BIND server on your home network.

The alternate is to use one of the external third-party services (OpenDNS or DNS Advantage). But these services also provide Phishing and Filtering services, which means that they can easily modify and redirect an incoming request using the most basic and critical service on the Internet.

While this may sound like the rant of a paranoid, it is a concept that has practical consequences. As an organization, how do you know that you aren’t on the DNS filter list of these providers, or the ISPs? If they can filter and redirect DNS requests, what else are they doing with the information? Are they providing open and trusted access to the core DNS services of the Internet?

Stepping back as far as you can into the process of going to your favorite pages, you will find that you can’t get there without DNS. And if DNS can no longer be trusted, even from legitimate providers, the entire basis of the Internet dissolves.

This afternoon, the two GrabPERF Agents at Technorati were switched back to using their local copies of caching BIND for resolving DNS entries.

Some folks at Microsoft who stopped by to look at their results on the Search Performance Index noticed that there were one or two outliers in the results from these locations. When I investigated, the OpenDNS name servers I was using were returning odd results.

The reason I had been using OpenDNS is that the local BIND instances were seeing unusual behavior a few months back. So, consider the switch to the local BIND instances probational, pending ongoing review.

Thanks to Eric Schurman for letting me know.

For a number of years, I have owned three very popular domain names: WebPerformance.org, WebCaching.org, and WebCompression.org. Last night, after many days of consideration, I stopped pointing them at their own distinct Web space and pointed them at this blog.

This is not a bad or evil thing, considering that for at least 18 months, the articles that were hosted at those sites were duplicated here in a more manageable format.

For those who have come looking for the content from those sites, it is here. The search box in the right column can help you locate it.

But for those who would like a refresher, here is a list of the most popular articles on this blog, as selected by Web performance traffic.

Web Performance Concepts Series

• Web Performance, Part I: Fundamentals
• Web Performance, Part II: What are you calling ‘average’?
• Web Performance, Part III: Moving Beyond Average
• Web Performance, Part IV: Finding The Frequency
• Web Performance, Part V: Baseline Your Data
• Web Performance, Part VI: Benchmarking Your Site
• Web Performance, Part VII: Reliability and Consistency
• Web Performance, Part VIII: How do you define fast?
• Web Performance, Part IX: Curse of the Single Metric

Why Web Measurement Series

• Why Web Measurement, Part I: Customer Generation
• Why Web Measurement, Part II: Customer Retention
• Why Web Measurement, Part III: Business Operations
• Why Web Measurement, Part IV: Technical Operations

Web Performance Tuning

• The Dichotomy of the Web: Andy King’s Website Optimization
• Performance Improvement From Compression
• Baseline Testing With cURL
• Compressing Web Output Using mod_deflate and Apache 2.0.x
• Compressing PHP Output
• Using Client-Side Cache Solutions And Server-Side Caching Configurations To Improve Internet Performance
• Performance Improvement From Caching and Compression
• Compressing Web Output Using mod_gzip for Apache 1.3.x and 2.0.x
• mod_gzip Compile Instructions
• Hacking mod_deflate for Apache 2.0.44 and lower

IP Tracking

• Geographic IP database using PERL, PHP and MySQL — UPDATE: September 16 2008
• Hit Tracking with PHP and MySQL

Budgets are shrinking. Resources are tight or shrinking. In a recent post, I discussed how ideas that I had been a proponent of for 2-3 years suddenly became extremely valuable to companies during the downturn of 2001-2003.

This downturn is a different beast. This means that you will need more than basic technical smarts to get through. To survive in the world of e-commerce for the next 18 months, here are some strategies you should take to heart.

• Clean up the cruft. Development and infrastructure changes are going to slow during the downturn – accept this. So do a code audit. Make sure that your applications aren’t calling files that no longer exist. Get rid of (backup and archive or Sarbanes-Oxley will bite your butt) old directories or application code that is no longer supported or maintained. In other words, do your Spring cleaning.

• Take care of the annoyances. In the rush to get code out or fix issues, little things that could help your site are often overlooked. Take the time now to optimize your images, minify your JS files, ensure your CSS files only contain styles that you use. Tune your SQL queries. Optimize your app code. All those things that got lost in the rush to get stuff out the door. As Microsoft and Apple have realized, people want performance, not more features.

• Make your site Browser Neutral. I work for a firm that allows me to view various sites with different OS/browser combinations easily and recommend to clients that they do the same. I hang my head when I look at my own blog and realize that it is not Browser Neutral. Take some time to investigate to how to eliminate this frustrating annoyance that makes people cranky and doubt your technical savvy.

• Get to know your visitors/customers. Unlike a ‘real’ store, you can’t step out onto the floor and talk to customers. But you can do this far more easily in today’s world than ever before. In the past, visitor analytics would have been the extent to which a company would have gone to determine information about their visitors/customers. Today, Twitter, Facebook, and all other manner of social sites make listening and talking to customers much easier. Just remember to be real.

• Get to know the people you work with. Just like getting to know your visitors/customers, you need to spend some time getting to know people in your organization. I know this sounds like manager-speak, but if you have contacts in Finance, Product Management, Operations, Admin, you are more likely to be able to more effectively due your job. If you understand the ebb and flow and stresses that are going on outside your little enclosed silo, you can place things in a larger perspective. While social-networking may work here, be careful to back it up with face-to-face contact.

• Get to know people! I made a number of contacts in the last downturn by taking my key interests (Web compression, Web caching) and turning them into sites, published articles, and one failed conference appearance (contact me if you want to hear more). Over the years these contacts have had me comment on their posts, edit their books, and keep them up-to-date on where I think the latest trends are headed. Going into this downturn, I have a whole group of new contacts that I am building on to do the same things. People aren’t bad, they’re just misunderstood.

• Read the fine business plan. I know these things are evil. They are in horrible management-speak. So don’t read it that way: Active read it. Make notes in the columns. Turn it into a MindMap (that’s what I did this year). Extract the meaning of what the goals of the business are and how the Web site fits into it. If you understand what the rest of the business is doing and what their challenges are, your problems will have context!

• Know what the next big thing is. While the objective of the organization in a downturn is to try and be more efficient with fewer resources, remember that you have to take time (whatever you have left) to know what’s coming next that could affect your Web site. Read the tech news. Understand the fundamentals that are driving the latest ideas. Keep on top of security. Sometimes a side-project that uses a technology that is new can become a bigger part of your site when resources free up.

I know these are general, but when this downturn started, I looked back on what worked for me during the last downturn. These are a few of the things that got me through, kept me busy, and helped me make an impact on the company and industry I am in.

These things also don’t hurt your reputation, as you become known as someone who understands the industry, the business, and the customers.

When starting with new clients, finding the low-hanging fruit of Web performance is often the simplest thing that can be done. By recommending a few simple configuration changes, these early stage clients can often reap substantial Web performance improvement gains.

The harder problem is that it is hard for organizations to build on these early wins and create an ongoing culture of Web performance improvement. Stripping away the simple fixes often exposes deeper, more base problems that may not have anything to do with technology. In some cases, there is no Web performance improvement process simply because of the pressure and resource constraints that are faced.

In other cases, a deeper, more profound distrust between the IT and Business sides of the organization leads to a culture of conflict, a culture where it is almost impossible to help a company evolve and develop more advanced ways of examining the Web performance improvement process.

I have written on how Business and IT appear, on the surface, to be a mutually exclusive dichotomy in my review of Andy King’s Website Optimization. But this dichotomy only exists in those organizations where conflict between business and technology goals dominate the conversation. In an organization with more advanced Web performance improvement processes, there is a shared belief that all business units share the same goal.

So how can a company without a culture of Web performance improvement develop one?

What can an organization crushed between limited resources and demanding clients do to make sure that every aspect of their Web presence performs in an optimal way?

How can an organization where the lack of transparency and the open distrust between groups evolve to adopt an open and mutually agreed upon performance improvement process?

Experience has shown me that a strong culture of Web performance improvement is built on three pillars: Targets, Measurements, and Involvement.

Targets

Setting a Web performance improvement target is the easiest part of the process to implement. it is almost ironic that it is also the part of the process that is the most often ignored.

Any Web performance improvement process must start with a target. It is the target that defines the success of the initiative at the end of all of the effort and work.

If a Web performance improvement process does not have a target, then the process should be immediately halted. Without a target, there is no way to gauge how effective the project has been, and there is no way to measure success.

Measurements

Key to achieving any target is the ability to measure the success in achieving the target. However, before success can be measured, how to measure success must be determined. There must be clear definitions on what will be measured, how, from where, and why the measurement is important.

Defining how success will be measured ensures transparency throughout the improvement process. Allowing anyone who is involved or interested in the process to see the progress being made makes it easier to get people excited and involved in the performance improvement process.

Involvement

This is the component of the Web performance improvement process that companies have the greatest difficulty with. One of the great themes that defines the Web performance industry is the openly hostile relationships between IT and Business that exist within so many organizations. The desire to develop and ingrain a culture of Web performance improvement is lost in the turf battles between IT and Business.

If this energy could be channeled into proactive activity, the Web performance improvement process would be seen as beneficial to both IT and Business. But what this means is that there must be greater openness to involve the two parts of the organization in any Web performance improvement initiative.

Involving as many people as is relevant requires that all parts of the organization agree on how improvement will be measured, and what defines a successful Web performance improvement initiative.

Summary

Targets, Measurements, and Involvement are critical to Web performance initiatives. The highly technical nature of a Web site and the complexities of the business that this technology supports should push companies to find the simplest performance improvement process that they can. What most often occurs, however, is that these three simple process management ideas are quickly overwhelmed by time pressures, client demands, resource constraints, and internecine corporate warfare.

Steven Hodson of WinExtra posted a screenshot of his personal Wordpress stats for the last three years last night. I then posted my stats for a similar period of time, and Steven shot back with some question about traffic, and the ebbs and flows of readers.

Being the stats nut that I am, I went and pulled the data from my own tracking data, and came up with this.

I made a conscious choice to analyze what year the posts being read were posted in. I wanted to understand when people read my content, which content kept people coming back over and over again. The chart above speaks for itself: through most of the last year it’s clear that the most popular posts were made in 2005.

What is also interesting is the decreasing interest in 2007 posts as 2008 progressed. Posts from 2006 remained steady, as there are a number of posts in that year that amount to my self-help guides to Web compression, mod_gzip, mod_deflate, and Web caching for Web administrators.

This data is no surprise to me, as I posted my rants against Gutter Helmet and their installation process in 2005. Those posts are still near the top of the Google search response for term “Gutter Helmet”. And improving the performance of a Web site is of great interest to many Apache server admins and Web site designers.

What is also clear is that self-hosting my blog and the posting renaissance it has provoked has driven traffic back to my site.

So, what lessons did I learn from this data?

1. Always remember the long tail. Every blogger wants to be relevant, on the edge, and showing that they understand current trends. The people who follow those trends are a small minority of the people who read blogs. Google and other search engines will expose them to your writings in the time of their choosing, and you may find that the three year-old post gets as much traffic as the one posted three hours ago
2. Write often. I was in a blogging funk when my blog was at Wordpress.com. As a geek, I believe that the lack of direct control over the look and feel of my content was the cause of this. In a self-hosted environment, I feel thta I am truly the one in charge, and I can make this blog what I want.
3. Be cautious of your fame. If your posts are front-loaded, i.e. if all your readers read posts from the month and year they are posted in, are you holding people’s long-term attention? What have you contributed to the ongoing needs of those who are outside the technical elite? What will drive them to keep coming to your site in the long run?

So, I post a challenge to other bloggers out there. My numbers are miniscule compared to the blogging elite, but I am curious to get a rough sense of how the long tail is treating you.

David Cancel and I have had sort of a passing vague, same space and thought process, living in the same Metropolitan area kind of distant acquaintance for about the same year.

About 2-3 months ago, he wrote a pair of articles discussing the efforts he has undertaken in order to try and offload some of the traffic to the servers for his new company, Lookery. While they are not current, in the sense that time moves in one direction for most technical people, and is compressed into the events of the past eight hours and the next 30 minutes, these articles provide an insight that should not be missed.

These two articles show how easily a growing company that is trying to improve performance and customer experience can achieve measureable results on a budget that consists of can recycling money and green stamps.

Measuring your CDN

A service that relies on the request and downloading of a single file from a single location very quickly realizes the limitations that this model imposes as traffic begins to broaden and increase. Geographically diverse users begin to notice performance delays as they attempt to reach a single, geographically-specific server. And the hosting location, even one as large as Amazon S3, can begin to serve as the bottleneck to success.

David’s first article examines the solution path that Lookery chose, which was moving the tag, which drives the entire opportunity for success in their business model, onto a CDN. With a somewhat enigmatic title (Using Amazon S3 as a CDN?), he describes how the Lookery team measured the distributed performance of their JS tag using a free measurement service (not GrabPERF) and compared various CDNs against the origin configuration that is based on the Amazon S3 environment.

This deceptively simple test, which is perfect for the type of system that Lookery uses, provided that team with the data they needed to realize that they had made a good choice in choosing a CDN and that their chosen CDN was able to deliver improved response times when compared to their origin servers.

Check your Cacheability

Cacheability is a nasty word that my spell-checker hates. To define it simply, it refers to the ability of end-user browsers and network-level caching proxies to store and re-use downloaded content based on clear and explicit caching rules delivered in the server response header.

The second Article in David’s series describes how, using Mark Nottingham’s Cacheability Engine, the Lookery team was able to examine the way that the CDNs and the Origin site informed the visitor browser of the cacheability of the JS file that they were downloading.

Cacheability doesn’t seem that important until you remember that most small firms are very conscious of the Bandwidth outlay. These small startups arevery aware when their bandwidth usage reaches 250GB/month level (Lookery’s bandwidth usage at the time the posts were written). Any method that can improve end-user performance while stilll delivering the service they expect is a welcome addition, especially when it is low-cost to free.

In the post, David notes that there appears to be no way in their chosen CDN to modify the Cacheability settings, an issue which appears to have been remedied since the article went up [See current server response headers for the Lookery tag here].

Conclusion

Startups spend a lot of time imagining what success looks like. And when it comes, sometimes they aren’t ready for it, especially when it comes to the ability to handle increasing loads with their often centralized, single-location architectures.

David Cancel, in these two articles, shows how a little early planning, some clear goals, and targeted performance measurement can provide an organization with the information to get them through their initial growth spurt in style.

The Web is a many-splendored thing, with a very split personality. One side is drive to find ways to make the most money possible, while the other is driven to implement cool technology in an effective and efficient manner (most of the time).

Andy King, in Website Optimization (O’Reilly), tries to address these two competing forces in a way that both can understand. This is important because, as we all know from our own lives, most of the time these two competing parts of the same whole are right; they just don’t understand the other side.

I have seen this trend repeated throughout my nine years in the Web performance industry, five years as a consultant. Companies torn asunder, viewing the Business v. Technology interaction as a Cold War, one that occasionally flares up in odd places which serve as proxies between the two.

Website Optimization appears at first glance to be torn asunder by this conflict. With half devoted to optimizing the site for business and the other to performance and design optimization, there will be a cry from the competing factions that half of this book is a useless waste of time.

These are the organizations and individuals who will always be fighting to succeed in this industry. These are the people and companies who don’t understand that success in both areas is critical to succeeding in a highly competitive Web world.

The first half of the book is dedicated to the optimization of a Web site, any Web site, to serve a well-defined business purpose. Discussing terms such as SEO, PPC, and CRO can curdle the blood of any hardcore techie, but they are what drive the design and business purpose of a Web site. Without a way to get people to a site, and use the information on the site to do business or complete the tasks that they need to, there is no need to have a technological infrastructure to support it.

Conversely, a business with lofty goals and a strategy that will change the marketplace will not get a chance to succeed if the site is slow, the pages are large, and design makes cat barf look good. Concepts such HTTP compression, file concatenation, caching, and JS/CSS placement drive this side of the personality, as well as a number of application and networking considerations that are just too far down the rat hole to even consider in a book with as broad a scope as this one.

Although on the surface, the concepts discussed in this book will see many people put it down as it isn’t business or techie enough, those who do buy the book will show that they have a grasp of the wider perspective, the one that drives all successful sites to stand tall in a sea of similarity.

See the Website Optimization book companion site for more information, chapter summaries and two sample chapters.

Port80 Software is reporting that in their survey of Fortune 1000 Web sites, IIS 6.0 has overtaken Apache as the Web server platform of choice. [here]

My two-cents: I respect the Port80 Software team greatly and love their maniacal devotion to ensuring that IIS users actually make use of the HTTP compression and caching that can so greatly improve Web performance.

That said, they are tied to Microsoft and the IIS platform. I would be curious to see if, scratching below the surface, they were able to determine what the application platform these companies built their mission critical Web applications on. I am open-minded and willing to hear that IIS is winning in that area as well. In my mind, it’s about Web performance tuning, not what you use to get that performance.

That said, I think a critical Web application survey of these same firms would find that many of these companies rely on JSP servers to run their core business processes.

As well, it would be interesting to se, by Fortune 1000 ranking, what the companies are using what server platform.

And…people still use Netscape Enterprise, SunOne, and Domino as production Web servers? YIKES!

This paper is an extension of the work done for another article that highlighted the performance benefits of retrieving uncompressed and compressed objects directly from the origin server. I wanted to add a proxy server into the stream and determine if proxy servers helped improve the performance of object downloads, and by how much.

Using the same series of objects in the original compression article[1], the CURL tests were re-run 3 times:

1. Directly from the origin server
2. Through the proxy server, to load the files into cache
3. Through the proxy server, to avoid retrieving files from the origin.[2]

This series of three tests was repeated twice: once for the uncompressed files, and then for the compressed objects.[3]

As can be seen clearly in the plots below, compression caused web page download times to improve greatly, when the objects were retrieved from the source. However, the performance difference between compressed and uncompressed data all but disappears when retrieving objects from a proxy server on a corporate LAN.

Instead of the linear growth between object size and download time seen in both of the retrieval tests that used the origin server (Source and Proxy Load data), the Proxy Draw data clearly shows the benefits that accrue when a proxy server is added to a network to assist with serving HTTP traffic.

	MEAN DOWNLOAD TIME
Uncompressed Pages
Total Time Uncompressed — No Proxy	0.256
Total Time Uncompressed — Proxy Load	0.254
Total Time Uncompressed — Proxy Draw	0.110
Compressed Pages
Total Time Compressed — No Proxy	0.181
Total Time Compressed — Proxy Load	0.140
Total Time Compressed — Proxy Draw	0.104

The data above shows just how much of an improvement is gained by adding a local proxy server, explicit caching descriptions and compression can add to a Web site. For sites that do force a great of requests to be returned directly to the origin server, compression will be of great help in reducing bandwidth costs and improving performance. However, by allowing pages to be cached in local proxy servers, the difference between compressed and uncompressed pages vanishes.

Conclusion

Compression is a very good start when attempting to optimize performance. The addition of explicit caching messages in server responses which allow proxy servers to serve cached data to clients on remote local LANs can improve performance to even a greater extent than compression can. These two should be used together to improve the overall performance of Web sites.

[1]The test set was made up of the 1952 HTML files located in the top directory of the Linux Documentation Project HTML archive.

[2]All of the pages in these tests announced the following server response header indicating its cacheability:

Cache-Control: max-age=3600

[3]A note on the compressed files: all compression was performed dynamically by mod_gzip for Apache/1.3.27.