In the GigaOm blog today, Allen Leinwand puts up a monstrous wake-up call to all the hip and cool Web 2.0 companies out there: Your apps run across the Internet [here].

I have spent 9 years investigating, diagnosing, and validating the Web performance issues of companies. I can tear the Web performance data of a site down quickly and ask pointed questions about why certain components of an application are behaving poorly.

But even after 9 years, there are still gimme problems around connection setup that I can seem brilliant about, not because I have some secret knowledge, but because I think of Web performance from the Network UP, not from the Application DOWN.

The subtlety of this difference what Leinwand is alluding to. Fancy applications run across the Internet. The Internet is built on TCP. And TCP is built on-top of a very complex networking infrastructure that is way beyond the realm of my skills.

If you don’t know what packet loss looks like, or how your fancy app presents to clients, or how to ensure that this data is collected and presented to you in a timely way, then you are being exposed to alerting by client calls.

All because you thought the biggest problem was scaling your app, not ensuring that the network it crossed to reach people affected the way it performed. Network geeks created Web 1.0; Web 2.0 seems to think they are mostly unecessary.

Wrong.

Measure your performance. Understand TCP. Hire a network geek (or 20).

Then sleep better at night.

So, Microsoft is releasing Internet Explorer 8 later this year, and a Beta is available now.

The question is, are you ready for it?

Internally, the tech savvy folks (myself included) have been tossing around the football of how to tackle it. The leap from 2 to 6 connections per host, on top of the basic rendering challenges that come with any new browser release are enough to make a Web team’s collective hive mind melt.

So, the bright team at Gomez are developing an IE 8 readiness kit.

Sounds like a great idea.

Why do you need to be concerned at a network level?

Let’s see…Well, while 6 simultaneous connections may be faster for clients, what happens to the TCP stacks of the devices that handle all of these connections, especially as more and more people start using IE 8?

Imagine that every Firefox user is using FasterFox, and then IE 8 comes along.

Do you have enough TCP sockets?

How quickly are sockets that are ready to be recycled actually recycled?

Are you really ready?

I am writing up a client presentation for next week, and I just realized just how flawed Internet Explorer is. Microsoft claims that the browser is standards compliant. Yet it still doesn’t support HTTP pipelining.

And the frustrating part? They won’t tell us why. I have my suspicions, which include TCP stack issues and a flawed HTTP handling mechanism that is still based on Windows 95 architecture, but an explanation from Redmond would be nice.

Every (and I mean every) other browser can do this.

Microsoft, it’s time you detached your Web browser from your OS, like you’ve forced everyone else to do.

Tags: HTTP, pipelining, RFC 2616, Firefox, Opera, HTTP Pipelining, Internet Explorer

Dear Flickr:

I have been wondering for sometime why downloads from your site seemed a little sluggish at times.

At first I blamed your unprecedented growth and success. For a little Vancouver startup (I am a BC boy myself), your entrance onto the stage of social networking applications has been phenomenal. The move from zero to infinity may have played a part in the performance I was seeing.

Nope. There was something else going on; I could see it every time I loaded a Flickr page in my browser. There was something else going on.

So today, I checked something out, and found the problem.

You need to enable persistent TCP connection on the static.flickr.com servers.

Now, that is the simple answer. I know that with large, web-based applications, enabling something as monumental as persistent connections could cause serious issues. If the architecture of the system was not designed to handle persistent connections, turning them on could cause the entire system to collapse.

There are legitimate, if mis-guided, reasons for disabled persistent connections. Some administrators believe that it is actually more efficient to have a client open a connection for every object. Easier to manage state, etc. The only problem is that in order to do that, you have to tune the systems serving data to shorten the amount of time a closed connection spends in a TIME_WAIT state.

When a TCP connection is closed, the socket is not immediately closed by the system in a default configuration. The TIME_WAIT state is the holding pen that these connections are pushed into. While in this state, the socket is locked and this may count against the incoming TCP connection queue, forcing the network stack to delay or reset new incoming connections.

Still, as Flickr is a worldwide company, the delay that the lack of persistent connections injects is astounding for locations in Asia. If you want to grow your business, and support more services, this will likely become a bottleneck very quickly.

Have a great weekend!

smp

Technorati Tags: Flickr, persistent connections, TCP, http, web performance, scalability

The gang at the Never Work Alone blog have a fantastic post describing some of the solutions to the Introverted IT / Extroverted Sales-Marketing integration issue.[here]

The best points:

• When hiring, place a premium on being able to explain technical issues to users and determine whether they’ve mastered the material. Expect this to cost more.
• Offer raises for taking training in oral technical communication
• Offer “days off” learning the essential business function of the department. You don’t understand what they do, they often don’t really GET what you do either, nor why its important – gieve them a chance to understand each other
• Train non-IT staff to repeat back in their own words what the IT person explained to them and confirm that they got it right (a good idea for any complex communication)

My eternal salvation comes from falling into the first category listed above. I can tear apart a packet trace and spot issues at the TCP layer, and then turn around and explain this issue to the VP of Marketing in terms that she can understand, and are relevant to her.

That is not dumbing it down, as many IT people feel. This strategy (or survival mechanism) allows a technical person to appeal to a wider audience. Being recognized across your organization, not just in your team, leads to greater rewards in the long run.

So, I mentioned earlier that I have packet loss on the uplink from my Web server where GrabPERF and this blog are hosted. How do I know it’s packet loss and not some other issue?

CLICK THE IMAGE

Notice the banding of measurements around 3 and 9 seconds? These values are the set TCP re-transmission timeouts for new connections for almost every operating system. When you see this pattern, check your packet loss stats from your routers/switches/network equipment.

Now…I wonder what my connectivity provider has mucked up?

You have noted that you are experiencing some performance issues related to high load (here). So I investigated and found that all the servers at the hostname www.technorati.com are responding with HTTP/1.0 headers and are explicitly closing the connections to the clients.

Why?

This will not relieve the performance problems. In fact, doing this may make the situation worse. The only way that this will not cause an issue is if you have tuned the kernel on these servers to go through an EXTREMELY fast TCP teardown process on the server side.

If you haven’t done this, go run a netstat -vanp on your www servers. See all the fin_wait, fin_wait2, closing and time_wait states? These are sockets that can’t be used again until the kernel releases them.

Now, Turning on keep-alives or persistent connections on the www servers will:

1. Reduce the total number of connections per client
2. Reduce the number of sockets “hung” in the teardown process
3. Improve performance by reducing the network overhead required by the client and the server

The only caveat is to reduce the keep-alive timeout to something like 4-5 seconds so that these connections don’t wait for traffic forever.

Improving performance through disabling persistent connections is a myth. HTTP/1.1 was adopted for a reason. Use it wisely and your will reap the rewards.

Technorati: web performance, HTTP, Technorati, Apache, PHP

Ok, after this morning’s DDoS, I started rummaging around for ways to limit the amount of hurt that my server would handle. And I found the limit function in iptables.

/sbin/iptables -A INPUT -p tcp -d 10.125.1.250 \
      --dport 80 -m limit --limit 6/m --limit-burst 10 -i eth0 -j ACCEPT
/sbin/iptables -A OUTPUT -p tcp -s 10.125.1.250 \
      --sport 80 -m limit --limit 6/m --limit-burst 10 -o eth0 -j ACCEPT

This should help get some of the requests under control.

Also, I discovered this interesting application called tc. Going to see how I can integrate this with some iptables rules.

Technorati: iptables

This morning, my server was the victim of a sustained DDoS lasting approximately 45 minutes. The entire flow of traffic came from the usual group of trackback and comment spam morons.

Now, the good news: b2evolution came through the event with flying colours. The antispam feature built into the product prevented ANY attempts by these morons at inserting comments and trackbacks from being successful.

I have added one more layer filtering to handle these morons. Since they use such a limited number of keywords in their REFERER fields, I just wrote a mod_rewrite rule to send them off to my infamous TCP Port 9080.

RewriteCond %{HTTP_REFERER} .*(pharmacy|poker|casino|blackjack|cialis|viagra| \
     porn|nude|girls|drugs|sex|animal|holdem| \
     stud|hydrocodone|vicodin|slut|anal|xanax|video| \
     oxycontin|russia|-online|online-).*
RewriteRule ^.*$ http://www.newestindustry.org:9080/ [R,L,NS]

This should deal with 90% of the morons. If I missed any keywords, drop me a comment.

Technorati: iptables, mod_rewrite, comment spam, trackback spam, DDoS

I have been seeing these bursts of traffic, mainly from spambot morons, that have suddenly been crushing my server. The main cause: excessive database connections.

This was quickly remedied today when I changed all of the mysql_connect statements to mysql_pconnect statements. This allows PHP to use an existing connection to the MySQL database to serve requests from the same Apache child process.

Now the truly geeky among you are going “DOH! Wadda ya mean you were opening a new connection for every request?”. Well, believe it or not, I will bet you dollars to doughnuts that your blog app doesn’t persist database connections. Not a big deal if your database is on the same machine, and you are using local named pipes to make requests. However, if that database is located on another machine, if you do a netstat, you will see a large number of connection on port 3306.

Persisting database connections is particularly important for large hosted services. A great deal of TCP overhead, and kernel space memory can be saved by simply not letting the Web server saturate the database with individual database connections for every page request.

Without persistent database connections, eventually the TCP queue will be full of database connections and no one will be able to connect to the server, or they will get a lovely “can’t connect to database error”.

Technorati: a href=”http://www.technorati.com/tag/web+performance” rel=”tag”>web performance, PHP, MySQL, MySQL PHP, Apache