Lately, there have been outages for two large sites: Amazon and Facebook. Working for a company that monitors such things made me able to confirm the nature of the outages.  But how I became aware of them has had me thinking in new ways for the last few weeks.

I became aware of both of these outages through a combination of FriendFeed and Twitter within minutes of them starting. This information spread quickly. And, due to the nature of these new technologies, people were able to comment on the outages, and theorize about the cause of the problems these large online firms faced.

The question you are likely asking is “So what?”. Well, as anyone who has been paying attention for the last four years should know, while you cannot completely control the conversation, you can participate in it and help prevent the spread of negative or incorrect theories about what is happening on your site.

The technologies that people who come to your site use to comment when something goes wrong can be used to interact with the customers. The classic example of this is Zappos. If you look on Twitter, you will find a number of members of that organization who are using the service to interact with customers on a human level. And if you have a problem or question, you stand an excellent chance of getting a response from the CEO if you ask a question.

So, if your site experiences an issue or problem, how do you interact with customers? Or do you just hope they don’t notice?

Netcraft noted that Yahoo encountered a bit of a headache today. So I fired up my handy-dandy little performance system and had a look.

Although for an organization and infrastructure the size of Yahoo’s this may have been a big event, in my experience, this was a "stuff happens on the Internet" sort of thing.

Move along people; there’s nothing to see. It is not the apocalyptic event that Netcraft is making it out to be. Google burps and barfs all the time, and everyone grumbles. But there is no need to run in circles and scream and shout.

Yeesh!

Tags: GrabPERF, Netcraft, Yahoo, outage, performance issues, panic, crisis

This morning, after months of increasing performance issues, and connectivity issues, I have retired the Boston Verizon measurement location. This location hosted 2 measurement agents.

The machines, hosted in my basement, are connected using Verizon FiOS, which has become increasingly flaky over the last couple of months. As well, the machines are 7 year old Pentium IIs, and require a substantial amount of spoon-feeding that I don’t have time for at the moment.

I have re-enabled the Technorati #1 Agent to fill the gap, but this leaves only 4 measurement locations running. I again put out my plea for more volunteers to host GrabPERF measurement locations. I have had one volunteer contact me about this (thanks Henrik!), and this location should be up in about 3 weeks.

If you have a spare linux box and a static IP, have I got a project for you!

PS: The contact page is fixed. It was set to auto-refresh and overwrite your e-mails. Ooooops.

Tags: GrabPERF, Agent, Verizon, Web Performance, measurement, performance measurement

Ensight and Blog Herald both discuss the floundering and thrashing of Technorati.

My take is that they are having a success failure.

© 2005 GrabPERF

This is how long it takes their servers to deliver on a simple search for “new york”. I have discussed at various points throughout the last week that there are some steps that Technorati could take to try and resolve their Web performance issues.

Technorati Search Content is still, for the most part, relevant for my needs. But I use Google most of the time, and just changed my Pinger to hit Ping-O-Matic to get into more search engines. Also a lot of good talk about IceRocket’s Blog Search tool.

If the Technorati team moves toward selling their services to businesses and corporations (a blog monitoring service), they are definitely putting the cart before the horse. Abandoning your core business to try and appeal to a “broader market” very rarely works.

And why are we having this discussion at all?

Because, dammit, we actually thought Technorati was different.

UPDATE: Stephen Baker at BusinessWeek found this post. [here]

Technorati: web performance, HTTP, Technorati

You have noted that you are experiencing some performance issues related to high load (here). So I investigated and found that all the servers at the hostname www.technorati.com are responding with HTTP/1.0 headers and are explicitly closing the connections to the clients.

Why?

This will not relieve the performance problems. In fact, doing this may make the situation worse. The only way that this will not cause an issue is if you have tuned the kernel on these servers to go through an EXTREMELY fast TCP teardown process on the server side.

If you haven’t done this, go run a netstat -vanp on your www servers. See all the fin_wait, fin_wait2, closing and time_wait states? These are sockets that can’t be used again until the kernel releases them.

Now, Turning on keep-alives or persistent connections on the www servers will:

1. Reduce the total number of connections per client
2. Reduce the number of sockets “hung” in the teardown process
3. Improve performance by reducing the network overhead required by the client and the server

The only caveat is to reduce the keep-alive timeout to something like 4-5 seconds so that these connections don’t wait for traffic forever.

Improving performance through disabling persistent connections is a myth. HTTP/1.1 was adopted for a reason. Use it wisely and your will reap the rewards.

Technorati: web performance, HTTP, Technorati, Apache, PHP

There is a slight chance that I could be going to Germany at the end of the month, so I need some help.

I am looking for reader comments/suggestion on a solid, robust, multi-country power adapter for my emergency bag (I fly off to foreign countries to solve Web performance issues almost daily — NOT!).

So far I am looking at the Kensington or the Belkin. But I want to hear what you folks out in radio-land can contribute to my buying decision.

Thanks!

TECHNORATI: Travel, Business Travel, International Travel, Web/Tech, Europe

In writing the last post, I was thinking about what factors go into making the Web performance of a site "excellent". What defines in the minds of the sites users/customers/visitors/critics/competitors that the performance of a Web site is excellent?

These are usually judged by the standard factors:

• Usability
• Responsiveness
• Availability
• Traffic
• Reliability
• Security
• Clarity

But within the company itself, how is the performance of their Web site judged to be excellent?

Right now, most people use the external metrics mentioned above to determine excellence. However, it must be remembered that there are two other critical factors that need to be considered when managing a large IT infrastructure.

• Ease of Management. This is a metric that is often overlooked when determining if a Web site is excellent from an internal perspective. Often it is simply assumed that running a large IT infrastructure is incredibly complex; in most cases this is true. However, is it too complex too manage efficiently and effectively? How much time is spent finding the cause of problems as compared to resolving them?
• Cost of Operation. This is always a big one with me. I look at sites that are trying to squeeze as much performance and availability out of their sites as they can. At some point, the business has to step back and ask, "How much does another half-second of speed cost us?". When this context is placed around the "need for speed", it may open a few eyes.

When this two critical internal factors are combined with the raw external data that can be collected, collated and analyzed, some other ideas come to the forefront as KPIs in Web Performance Excellence:

• Cost Per Second. The cost of a Web site is usually only calculated based on the negative metric of how much it costs when the site is down. Well, how much does it cost when the site is up? Can that number be reduced?
• Revenue By Speed. Which customers spend the most on your site: LAN, home-broadband, or dial-up?
• Person-hours per day. How many person-hours per day does it take to manage your Web site?
• True Cost of Performance Issues. When there is a performance issue, the cost is usually associated with lost revenue. Reverse that and ask how much did it cost in time and materials to resolve the issue.

The creation of new Web performance excellence metrics is crucial if companies truly want to succeed in the e-business arena. Business management has to demand that IT management become more accountable to the entire business, using metrics that clearly display the true cost of doing business on the Web.