One of the traditional areas of frustration for Operations and Development teams in the Web world is that their performance, Web performance, is measured from the outside-in.

The resistance of this camp is strong, and they will appear without warning, even from amongst the most enlightened of companies.

How can they be recognized?

You will hear their battle-cry, their mantra, their fundamental belief that their application, their infrastructure is a misunderstood victim. That if they could only get their one idea across, the whole of the company would be enlightened.

The fundamental tenet of this group is simple and short.

How can we manage the Internet?

The obvious fallacy of this argument is clear to any Web performance professional or business analyst: Customers get to our business across the Internet, not via psychic modem. In order to keep close tabs on the experience of our customers, the site, application, code must be measured from the outside-in.

In order to prevent making enemies and perpetuating already ossified corporate silos, take the initiative. Gently steer the discussion in a new direction by making this incredibly vast problem into one everyone in the company can understand. By adding a single word to the initial question, the fearful and reactive perspective can be dramatically shifted to one that could make the members of this camp see the light.

Make the question:

How can we manage for the Internet?

Now the focus of the discussion is now proactive – is there something we are missing that could reduce the problems and/or prevent them from ever happening?

Taking the all-encompassing and awe-inspiring challenge that is the Internet and turning it into a Boy Scout moment may reinvigorate the internal conversation, and give people a sense of purpose. Now they will be galvanized to consider whether everything in their power is being done to prevent performance issues before bits hit the Internet.

Effective Web performance hinges on taking the obvious challenges that face all Web sites, and turning them into solutions that mitigate these challenges as much as possible. So, in the next team meeting, the next time you hear someone say that it’s just the Internet, ask what can still be done to manage the application more effectively for the Internet.

The moment a Web site goes live, the publishers lose control of the performance.

When I say lose control of the performance, I mean that despite everything that has been done to ensure scalability and capacity, the Web is inherently an infrastructure that is out of anyone’s direct ability to manage.

This is something that needs to be accepted. And while the datacenter is only that part of an application/infrastructure/network that can be directly managed by the Web site’s owners, a company has to accept that the real datacenter is the Internet. Not a datacenter that is on the Internet; the Internet as the datacenter.

Now that your head is spinning, let’s step back and consider this idea for a minute. The whole concept of the Internet being the datacenter makes operations and IT folks very uncomfortable. Why? There is no way for one company to manage the Internet. As a result, the general perspective is that the Internet can’t be trusted, and all that can be done is manage what can be managed directly.

Ignoring the Internet allows many organizations to leave the entire Internet out of their application or performance planning. They will measure and monitor, and they may even employ third-parties to help improve performance. When the shiny exterior is peeled back, it’s pretty clear that these organizations have built their entire performance culture on the assumption that if a problem exists on the Internet, there is nothing that can be done by them to fix it.

This may be effectively true. And it is not positive way to ensure effective Web performance

Having a what-if, emergency response plan in place is never a bad idea. If a problem appears on the Internet, and it affects your Web site, what are you going to do about it? Whine and moan and point fingers? Or take actions that effectively and clearly communicate to customers the steps you are taking to make things right?

Wait. Managing the Internet through customer communication?

I argue that besides working feverishly behind the scenes to resolve the problem, customer communication is the next most critical component of any Web performance issue management plan.

Web performance issue management plan. You have one, don’t you?

Well, when you get around to it, here are some concepts that should be built into the plan.

Effectively monitor your site

How can measurement and monitoring be part of issue management? Well, isn’t it always good policy to detect and begin investigating problems before your customers do?

Key to the measurement plan is monitoring the parts of your application that customers use. A homepage test will not give you vital information on issues with your authentication process, and is the same as saying the car starts, while ignoring the four flat tires.

If you aren’t effectively monitoring your site, your business is at risk.

Measure where the customers are

If your organization is focused on what it can control, then it will want to measure from locations that are controlled, and can provide stable, consistent, repeatable data.

Hate to break this to you, Sparky, but my Internet connection isn’t an OC-48 provisioned through a large carrier with a written SLA. Real people have provider networks that are congested, under-built, and deliver bandwidth using the old best effort approach.

Some customers may have given up on wires altogether, and access the site through wireless broadband or mobile devices.

Understand how your customers use your site. Then plan your response to managing the Internet from the outside-in.

Test with what your customers use

The greatest cop-out any Web site can make is Our site is best viewed using…

I’m sorry. This isn’t good enough.

Customers demand that your site work the way they want it to, not the other way around. If a customer wants to use Safari on a Mac, or Chromium on Linux, then understanding how the site performs and responds with these browsers is critical.

The one-browser/one-platform world no longer exists. If a large number of customers with one particular configuration indicate that they are having a problem with the new site, what is the proper reaction?

And why did this happen in the first place?

Monitor and respond to social media

No, this isn’t just here for buzzwords and SEO. In the last year, Twitter and Facebook have become the de-facto soapboxes for people who want to announce that their favorite site isn’t working. Wouldn’t hurt to monitor these sites for issues that might not be detected by traditional performance monitoring.

This approach means that you have to be willing to accept responsibility when something affects your site performance or availability, even if it isn’t your fault. No need to tell folks exactly what the problem is, but acknowledging that there is a legitimate issue that you recognize will go a long way toward making visitors/customers more understanding of the situation.

Get your message out effectively

Communicating about a performance issue means that the Marketing and PR teams will have to be brought in.

What? Marketing and Operations/IT working together? Yes. In a situation where there is a major outage or issue, Marketing will DEMAND to be involved. Wouldn’t it be easier if these two parts of the organization knew each other and a plan for responding to critical performance issues?

If Marketing understands the degree of the problem, what it will take to fix, and what is being done about it, they can craft a message that handles any question that might come in, while acknowledging that there is an issue.

A corollary to this: If there is an issue, don’t deny it exists. Denying a problem when it clear to anyone using the site that there is one is worse than saying nothing at all.

Takeaway

Practicing effective Web performance means a company understands that directly managing the Internet is impossible, but having a process to respond to Internet performance issues is critical. A Web performance incident plan shows that you understand that stuff happens on the Internet and you’re working on it.

This week marks a momentous time in the history of the Internet. In the United States, StatCounter reports that for the first three days of the work week (Monday – Friday), Internet Explorer 8 usage is equal to Internet Explorer 6 usage.

Tie this to the trend of decreasing Internet Explorer usage noted late last week and the release of Firefox 3.5 RC1 and Safari 4.0 and Opera 10 in the last few weeks, and it appears that the balance of browser usage on the Internet is becoming more fluid.

Does this mean that standards will become more relevant? Can you truly count on limiting customers to one browser?

Will browser lock-in for certain applications continue to be considered acceptable?

The idea that blogs generate a personal brand is as old as the “blogosphere”. It’s one of those topics that rages through the blog world every few months. Inexorably the discussion winds its way to the idea that a blog is linked exclusively to the creators of its content. This makes a blog, no matter what side of the discussion you fall on, the online representation of a personal brand that is as strong as a brand generated by an online business.

And just as corporate brands are affected by the performance of their Web sites, a personal brand can suffer just as much when something causes the performance of a blog Web site to degrade in the eyes of the visitors. For me, although my personal brand is not a large one, this happened yesterday when Disqus upgraded to multiple databases during the middle of the day, causing my site to slow to a crawl.

I will restrain my comments on mid-day maintenance for another time.

The focus of this post is the effect that site performance has on personal branding. In my case, the fact that my blog site slowed to a near standstill in the middle of the day likely left visitors with the impression that my blog about Web performance was not practicing what it preached.

For any personal brand, this is not a good thing.

In my case, I was able to draw on my experience to quickly identify and resolve the issue. Performance returned to normal when I temporarily disabled the Disqus plugin (it has since been reactivated). However, if I hadn’t been paying attention, this performance degradation could have continued, increasing the negative effect on my personal brand.

Like many blogs, Disqus is only one of the outside services I have embedded in my site design. Sites today rely on AdSense, Lookery, Google Analytics, Statcounter, Omniture, Lijit, and on goes the list. These services have become as omnipresent in blogs as the content. What needs to be remembered is that these add-ons are often overlooked as performance inhibitors.

Many of these services are built using the new models of the over-hyped and mis-understood Web 2.0. These services start small, and, as Shel Israel discussed yesterday, need to focus on scalability in order to grow and be seen as successful, rather than cool, but a bit flaky. As a result, these blog-centric services may affect performance to a far greater extent than the third-party apps used by well-established, commercial Web sites.

I am not claiming that any one of these services in and of themselves causes any form of slowdown. Each has its own challenges with scaling, capacity, and success. It is the sheer number of the services that are used by blog designers and authors poses the greatest potential problem when attempting to debug performance slowdowns or outages. The question in these instances, in the heat of a particularly stressful moment in time, is always: Is it my site or the third-party?

The advice I give is that spoken by Michael Dell: You can’t manage what you can’t measure. Yesterday, I initiated monitoring of my personal Disqus community page, so I could understand how this service affected my continuing Web performance. I suggest that you do the same, but not just of this third-party. You need to understand how all of the third-party apps you use affect how your personal brand performance is perceived.

Why is this important? In the mind of the visitor, the performance problem is always with your site. As with a corporate site that sees a sudden rise in response times or decrease in availability, it does not matter to the visitor what the underlying cause of the issue is. All they see is that your site, your brand (personal or corporate), is not as strong or reliable as they had been led to believe.

The lesson that I learned yesterday, one that I have taught to so many companies but not heeded myself, is that monitoring the performance of all aspects of your site is critical. And while you as the blog designer or writer might not directly control the third-party content you embed in your site, you must consider how it affects your personal brand when something goes wrong.

You can then make an informed decision on whether the benefit of any one third-party app is outweighed by the negative effect it has on your site performance and, by extension, your personal brand.

At the The China Vortex, Paul Denlinger discusses how there is no unified “China market”, no monolithic, simplistic, single-minded Goliath that the rest of the world is trying to deal with. While I do not have the depth of on the ground experience that Mr. Denlinger has (I have not yet been blessed with the opportunity to visit or do business in China), I can see the truth he brings to the discussion.

One of the great pits that Western culture falls into when dealing with the China problem is just that: It is seen as a problem, not an opportunity to expand and learn from a culture that deals with life, philosophy, and business in a very different manner.

This should come as no surprise to any astute student of History, or even modern geopolitics, as the way that nations deal with perceived threats or challenges is to create a national culture of The Other, the us-v-them foreign policy.

When Japan was the country du jour in the 1980s, the Western World respected it, in a very shallow way, as a fellow industrial nation with a strong warrior culture. However, it was treated in a simple way, with Western media portrayals that strengthened perceived stereotypes, and plastered over the profound differences that exist within Japan, and within the Japanese people.

China is even more of a victim of this Politics of the Other, having spent more than 50 years as one of the adversaries in the Cold War, being vilified and portrayed in the least flattering light possible. Even without the base Human interpretation of simplistic interpretations of the Other, the West is crippled from the start in its attempts to understand a nation as large, diverse, and fractured as China.

China is far more than Beijing, Shanghai, Hong Kong and small cadre of smaller, but no less important industrial / post-industrial metropolitan areas.

Drawing on my experience in trying to interpret Internet performance data from within this nation, it is clear to even the casual observer that the Chinese Internet does not simply exist in the major cities. It extends into the far reaches of the country, fractured by the internal conflicts of the connectivity providers, government officials at a many levels, and the unstoppable drive and creativity of the people who see the Internet as an opportunity to make their way in their world.

Cultural and national stereotypes are the way that humans ineffectively deal with the differences that exist. But just as the terms “All Brits..”, “All the French..”, “All Germans…”, “All Argentinians..”, et al. should be treated with disdain and seen as a sign of ignorance, using the words “All Chinese…” or “All of China…” should be quickly quashed and carted off to the dustbin of simplistic paranoia and xenophobia.

There is no such thing as a threat. As it is often stated in other contexts, a threat is simply an opportunity that is hidden by your own prejudices.

In the GigaOm blog today, Allen Leinwand puts up a monstrous wake-up call to all the hip and cool Web 2.0 companies out there: Your apps run across the Internet [here].

I have spent 9 years investigating, diagnosing, and validating the Web performance issues of companies. I can tear the Web performance data of a site down quickly and ask pointed questions about why certain components of an application are behaving poorly.

But even after 9 years, there are still gimme problems around connection setup that I can seem brilliant about, not because I have some secret knowledge, but because I think of Web performance from the Network UP, not from the Application DOWN.

The subtlety of this difference what Leinwand is alluding to. Fancy applications run across the Internet. The Internet is built on TCP. And TCP is built on-top of a very complex networking infrastructure that is way beyond the realm of my skills.

If you don’t know what packet loss looks like, or how your fancy app presents to clients, or how to ensure that this data is collected and presented to you in a timely way, then you are being exposed to alerting by client calls.

All because you thought the biggest problem was scaling your app, not ensuring that the network it crossed to reach people affected the way it performed. Network geeks created Web 1.0; Web 2.0 seems to think they are mostly unecessary.

Wrong.

Measure your performance. Understand TCP. Hire a network geek (or 20).

Then sleep better at night.

In today’s Boston Globe, there is an article discussing why Facebook went to the Valley instead of staying in the Boston area (article online).

Having now lived in both areas for nearly equal amounts of time, I can tell you that there are substantial differences between them. People from Boston may violently disagree, but I have found that the innovative spirit of the Valley, the one that drove the creation of the commercial Internet, does not exist here.

I am, however, someone who now laughs at the insular culture of the Valley, a place that still considers itself the center of the Internet innovation universe. I had a chance to meet with a growing Internet firm while I was out there on business last week (not Technorati), and I found the hubris and ego in the meeting that I attended laughable.

I was not laughing at this firm’s success, which has been great. I was laughing at the fact that the mid-level managers that we met with had the gall to effectively state that having their name on our customer list entitled (and yes, entitlement is also a large part of the culture) them to demand a deal that none of our customers get.

I can’t be sure what the sales guy I went with thought, but I left the meetings laughing. This company, which is younger than my youngest son, thought it had more pull with us than the multi-hundred billion financial firms we deal with daily. Thought that it had more pull than the large, first-generation Internet companies that we work closely with.

Yes, Boston does not generally fund and encourage a culture of innovation (yes, there are always exceptions). But those who seek to take the next great idea to the Internet should beware the hubris of the Valley.

My system has a daily job to collect and aggregate the IP Blocks distributed by the five registrars into a single database, and then provide high-level WHOIS information for this data. If you want to try this yourself, the interface here.

On an extremely irregular basis, I aggregate the statistics from this data, and present it to the masses for the examination. I might actually automate this data someday!

So, for August 2007 (as of August 21, 2007), here are the aggregated IP distribution statistics broken down by registrar and country.

Starting with the country data, I figured that a sane cutoff for values would be 1,000,000 IP addresses. The list here is mostly unchanged, except for the addition of the European Union (EU). The EU is a recent addition, as it was not listed in the Country Code lookup table I had been using, and had to be added manually on August 20, 2007.

COUNTRY
IPS


UNITED STATES
1394044928


JAPAN
156239616


CHINA
127511296


EUROPEAN UNION
120433884


UNITED KINGDOM
82641530


CANADA
73318912


GERMANY
69600688


FRANCE
65257760


KOREA, REPUBLIC OF
57733376


AUSTRALIA
32543744


ITALY
24079040


BRAZIL
21366272


SPAIN
20058528


MEXICO
19406848


NETHERLANDS
19265064


TAIWAN, PROVINCE OF CHINA
18736896


SWEDEN
16378336


RUSSIAN FEDERATION
15920408


INDIA
13495040


POLAND
12080076


SOUTH AFRICA
10817792


FINLAND
8577920


TURKEY
8287680


DENMARK
7872864


SWITZERLAND
7311136


HONG KONG
6909952


AUSTRIA
6634208


NORWAY
6611232


ROMANIA
5997696


BELGIUM
5419904


NEW ZEALAND
4971008


CZECH REPUBLIC
4791168


ARGENTINA
4749056


ISRAEL
4042176


CHILE
4041728


IRELAND
3851776


SINGAPORE
3821568


VIETNAM
3795712


THAILAND
3629568


VENEZUELA
3457536


PORTUGAL
3371840


HUNGARY
3312768


MALAYSIA
3133440


GREECE
2941440


COLOMBIA
2817280


UKRAINE
2446144


BULGARIA
2395648


PHILIPPINES
2328832


INDONESIA
2221056


EGYPT
2105088


LITHUANIA
1920384


SAUDI ARABIA
1572608


SLOVAKIA
1358080


IRAN, ISLAMIC REPUBLIC OF
1331712


COSTA RICA
1303296


UNITED ARAB EMIRATES
1230848


LATVIA
1161536


SLOVENIA
1060890


PERU
1023744

The next data is the IP addresses by registrar. Again, the order of this list is not surprising, except for the continued overwhelming dominance of ARIN.

REGISTRY
IPS


arin
1552509952


ripencc
497148492


apnic
406221568


lacnic
61819136


afrinic
16387328

I hope that this information is helpful and informative.

Technorati Tags: ARIN, RIPE, LACNIC, AFRINIC, APNIC, registrar, IP bloks, internet dominance, internet geography, statistics

Netcraft noted that Yahoo encountered a bit of a headache today. So I fired up my handy-dandy little performance system and had a look.

Although for an organization and infrastructure the size of Yahoo’s this may have been a big event, in my experience, this was a "stuff happens on the Internet" sort of thing.

Move along people; there’s nothing to see. It is not the apocalyptic event that Netcraft is making it out to be. Google burps and barfs all the time, and everyone grumbles. But there is no need to run in circles and scream and shout.

Yeesh!

Tags: GrabPERF, Netcraft, Yahoo, outage, performance issues, panic, crisis

My side project, GrabPERF, is looking for a few good measurement locations.

Right now, there are only five measurement locations, two of which are in my basement, on my personal Internet connection. I am hoping, through this pledge drive, to find a number of additional locations. Areas desperately needed include:

• East Coast, USA
• West Coast, USA
• Midwest, USA
• UK
• Asia-Pac
• Southeast Asia
• Australia / New Zealand

Yeah, I know. I am asking for the world. Can’t hurt to try though.

Basic requirements are a Linux box with a static IP address. Additional requirements are documented here.

You can express your interest in hosting a measurement site by filling in the GrabPERF Contact Form or contacting me directly.

Thank you for your continuing support.

Tags: GrabPERF, measurement locations, Web performance