It’s a rare Web site these days that hosts all of its own content. From the smallest blog to the largest retailer, Web sites farm out their images, streams, and pages to CDNs, and absorb feeds, ads, and data streams from any number of outside providers.

Effective Web performance demands that a site take responsibility for the entire site, not just the parts under direct local management. Why? Because customers see a problem with your site, not with a provider.

How can the performance all of the third-party content on a site be managed? Using the exactly the same strategies already place to manage the performance of local content.

Measure from the outside-in

Customers come from the Internet. That measuring the performance of a site from the perspective of visitors is being mentioned here should not be a surprise. Critical to this part of managing third-parties is the ability to see into the page and determine if there are performance issues requesting and transmitting data from third-parties.

In the first article of this series, I detailed a number of approaches to actively gathering performance data. This method, whether from the datacenter or the last mile, will provide the early warning signs that there is an issue with a third-party, and feeding this data into the performance issue management plan.

Measure from inside the browser

The network and application performance of a third-party page component is just the start of the process, as this is what it takes to get the object to the browser. But what if this object then launches a number of actions, or starts to render on the screen. This may lead to a whole different range of issues that are a blind spot when analyzing Web performance.

Measuring the performance of discrete page elements from within the visitors browser will provide deeper insight into what effects the customer sees and which third-parties will need to be approached in order to improve the overall Web performance of the site.

Have clear and useful SLOs and SLAs

Service level objectives and service level agreements are often thrown about whenever there is the suspicion that there is a Web performance issue. Using these documents and frameworks as a club to beat up partners with is counter-productive.

SLOs and SLAs should clearly detail:

• the performance expectations of the Web site owner
• the performance and delivery capabilities of the third-party provider

Guess what? Arriving at this in a way that doesn’t lead to resentment and mistrust on both sides requires open and honest discussion.

Share data

If Web site owners and third-parties are going to work together to ensure the most effective Web performance strategy possible, then data must flow freely. Vendors will need access to the same data that Web site owners have (and vice versa) in order to ensure that if an issue is detected, everyone can examine all of the available data, and solve the problem quickly.

Communication

A recurring and critical theme when establishing a culture of effective Web performance is communication. When working with third-parties, this is even more critical, as the performance culture of one organization may be completely different from another. The Web site owner may have one site of criteria that determines a Web performance issue, while the vendor has another, and unless these are understood, problems will occur.

Clear communication paths must be baked into the SLA. Named contacts or contact paths will be there, as will expected response times for inbound requests, and escalation procedures.

When there is a performance issue, both sides will need to be very clear about how each other will respond.

Takeaways

Third-party content on Web sites is a fact. It shouldn’t be a headache. Effective Web performance measurement strategies, shared sources of Web performance data, and clearly understood paths and methods of communication will make using third-party content less stress-inducing to everyone.

The moment a Web site goes live, the publishers lose control of the performance.

When I say lose control of the performance, I mean that despite everything that has been done to ensure scalability and capacity, the Web is inherently an infrastructure that is out of anyone’s direct ability to manage.

This is something that needs to be accepted. And while the datacenter is only that part of an application/infrastructure/network that can be directly managed by the Web site’s owners, a company has to accept that the real datacenter is the Internet. Not a datacenter that is on the Internet; the Internet as the datacenter.

Now that your head is spinning, let’s step back and consider this idea for a minute. The whole concept of the Internet being the datacenter makes operations and IT folks very uncomfortable. Why? There is no way for one company to manage the Internet. As a result, the general perspective is that the Internet can’t be trusted, and all that can be done is manage what can be managed directly.

Ignoring the Internet allows many organizations to leave the entire Internet out of their application or performance planning. They will measure and monitor, and they may even employ third-parties to help improve performance. When the shiny exterior is peeled back, it’s pretty clear that these organizations have built their entire performance culture on the assumption that if a problem exists on the Internet, there is nothing that can be done by them to fix it.

This may be effectively true. And it is not positive way to ensure effective Web performance

Having a what-if, emergency response plan in place is never a bad idea. If a problem appears on the Internet, and it affects your Web site, what are you going to do about it? Whine and moan and point fingers? Or take actions that effectively and clearly communicate to customers the steps you are taking to make things right?

Wait. Managing the Internet through customer communication?

I argue that besides working feverishly behind the scenes to resolve the problem, customer communication is the next most critical component of any Web performance issue management plan.

Web performance issue management plan. You have one, don’t you?

Well, when you get around to it, here are some concepts that should be built into the plan.

Effectively monitor your site

How can measurement and monitoring be part of issue management? Well, isn’t it always good policy to detect and begin investigating problems before your customers do?

Key to the measurement plan is monitoring the parts of your application that customers use. A homepage test will not give you vital information on issues with your authentication process, and is the same as saying the car starts, while ignoring the four flat tires.

If you aren’t effectively monitoring your site, your business is at risk.

Measure where the customers are

If your organization is focused on what it can control, then it will want to measure from locations that are controlled, and can provide stable, consistent, repeatable data.

Hate to break this to you, Sparky, but my Internet connection isn’t an OC-48 provisioned through a large carrier with a written SLA. Real people have provider networks that are congested, under-built, and deliver bandwidth using the old best effort approach.

Some customers may have given up on wires altogether, and access the site through wireless broadband or mobile devices.

Understand how your customers use your site. Then plan your response to managing the Internet from the outside-in.

Test with what your customers use

The greatest cop-out any Web site can make is Our site is best viewed using…

I’m sorry. This isn’t good enough.

Customers demand that your site work the way they want it to, not the other way around. If a customer wants to use Safari on a Mac, or Chromium on Linux, then understanding how the site performs and responds with these browsers is critical.

The one-browser/one-platform world no longer exists. If a large number of customers with one particular configuration indicate that they are having a problem with the new site, what is the proper reaction?

And why did this happen in the first place?

Monitor and respond to social media

No, this isn’t just here for buzzwords and SEO. In the last year, Twitter and Facebook have become the de-facto soapboxes for people who want to announce that their favorite site isn’t working. Wouldn’t hurt to monitor these sites for issues that might not be detected by traditional performance monitoring.

This approach means that you have to be willing to accept responsibility when something affects your site performance or availability, even if it isn’t your fault. No need to tell folks exactly what the problem is, but acknowledging that there is a legitimate issue that you recognize will go a long way toward making visitors/customers more understanding of the situation.

Get your message out effectively

Communicating about a performance issue means that the Marketing and PR teams will have to be brought in.

What? Marketing and Operations/IT working together? Yes. In a situation where there is a major outage or issue, Marketing will DEMAND to be involved. Wouldn’t it be easier if these two parts of the organization knew each other and a plan for responding to critical performance issues?

If Marketing understands the degree of the problem, what it will take to fix, and what is being done about it, they can craft a message that handles any question that might come in, while acknowledging that there is an issue.

A corollary to this: If there is an issue, don’t deny it exists. Denying a problem when it clear to anyone using the site that there is one is worse than saying nothing at all.

Takeaway

Practicing effective Web performance means a company understands that directly managing the Internet is impossible, but having a process to respond to Internet performance issues is critical. A Web performance incident plan shows that you understand that stuff happens on the Internet and you’re working on it.

Content Delivery Networks (CDNs) are a key component to any Web performance strategy. If you examine the content from any large online business or media provider, it won’t take long to find the objects that these organizations have entrusted to CDNs to ensure faster delivery and a better user experience.

When working with CDNs, it is critical to understand some terms or concepts that you will be presented with. Each CDN will present them in it’s own unique way and using its own unique terminology. Having an understanding of the underlying concepts, you will be able to have discussions with CDNs that are more meaningful, and targeted on your needs.

The Massively Distributed Model

CDNs fall into one of two categories, the first being the massively distributed model. CDNs that use this method will demonstrate how they have hardware and caching content servers in almost every city and town of any size in the world. As well, they have their systems located on every major consumer network in order to ensure that they are as close to the end-user as possible.

The CDN everywhere model, while far-reaching and seemingly extremely effective does have its disadvantages. First, the CDN infrastructure relies on having extremely accurate maps of the Internet in order to direct visitors to the most proximate CDN server location. However, these maps are only truly effective when visitors use DNS servers that are on the same network that they are. Services such as OpenDNS and DNS Advantage can seriously effect the proximity algorithms of the distributed CDN by removing the key piece of localization information that they need to ensure that the best cache location is selected.

Also, as with any proxy caching methodology, this model relies on use. More popular items stay in the cache longer, while less popular items may be pushed aside or stored further upstream at parent caches for retrieval, adding a few extra milliseconds for the initial request. Also, new content has to be pushed out to the edge, and may take a few hours to be completely propagated.

The Massively Concentrated Model

CDNs that use this model rely on a smaller number of locations than the massively distributed model. However, these locations tend to be massive and incredibly well connected, relying on the concept that even if they are a few more hops away, their content is always there and ready for requests.

These sites have massive amounts of storage and rely on private networks to ensure that new content is immediately pushed out to the super-nodes as soon as it is added. And while they may be those extra few hops away, the performance difference may not be enough for the average site visitor to notice.

The obvious disadvantage of the massively concentrated model is that it is great for serving those places where there is a lot of traffic. However, in regions with less traffic, or less developed infrastructures, the fewer boots on the ground may begin to have an effect on performance.

Other CDN Concepts

Application Proxy

CDNs offer many institutions the ability to use their network for all incoming requests, even if they are for dynamic content that will require processing in the client datacenter. In these instances, the CDN acts as an application proxy, using its superior knowledge of routing and traffic patterns to move requests from the edge of the Internet back to the datacenter more effectively.

Remember: Just because the CDN is providing fast routing and delivery to the visitor, your application is still the bottleneck. Poor app design or slow queries will affect the application in exactly the same way that it would if the call was coming straight to your datacenter.

Traffic Acceleration

In certain circumstances, security and regulatory concerns completely eliminate the ability of a business to use the standard CDN model. Banks, government agencies, and health-care providers cannot store data in an environment whose security they cannot vouch for, no matter how many safeguards are put in place.

These organizations still need to be able to deliver a good customer experience, so there has to be a way to help accelerate their content without taking control of it. Traffic acceleration serves this purpose by using proprietary network protocol adaptations that remove some of the overhead associated with standard network protocols.

Content is intercepted at the datacenter and routed across private networks using the streamlined network protocols to an network location that is as close to the visitor as possible. Once it has reached the appropriate location, it is converted back to standard TCP and passed to the visitor.

The method above describes how a standard Web request works, but this can also be extended to true point-to-point VPNs with endpoints separated by great network and/or physical distances.

Validating the Claims

Any component of choosing or using a CDN is quantifying the effectiveness of the solution. The standard for many years has been the bake-off method of comparison. The prospect’s origin site is measured against the same site delivered by one or more CDNs. The CDN vendor with the fastest performance and the best price usually wins.

Before walking into a bake-off, come prepared. Turn your CDN bake-off into an episode of Iron Chef. Come to the table with the ingredients, and make the CDNs prepare a solution that meets your needs.

Measure Transactions

The standard base measurement that CDNs will use in a bake-off is single object(s) or page measurement. Your visitors do not just visit a single page, so ensure that the CDN has an effective solution that produces noticeable performance improvements across all the key functions of your site, including the secure components of the site, where the money is made.

Measure from the Edge

Backbone measurements are great for baselining and detecting operational issues that require a consistent and stable dataset. Your customers, however, do not have direct connections to high-priced datacenters with fat pipes.

The two CDN models will react differently to under certain circumstances, and this will appear in edge measurements. Measuring on the ground, from the ISPs that your customers use, will give you a clear sense of how much improvement a CDN will provide when compared to the performance of your origin datacenter.

The edge is messy, chaotic, and what your customers deal with everyday.

Understand the SLAs/SLOs

CDNs will always provide either service level agreement (SLA) with service level objectives (SLOs) stated in it. This topic is at once recognizable and about as well understood as 11 Dimensional Theoretical Physics.

I have written briefly about SLAs and SLOs before [here and here]. Do your research before you wade into this polite version of white-collar trench warfare.

Make sure you understand what the goal of the SLA is. Make sure that the SLOs are clear, measurable, valid, and enforceable. Then ensure that the method used to measure the SLOs is one that your organization can understand and can accept as valid.

Finally, ensure that the SLOs are reviewed monthly.

Takeaways

Understanding the foundational technology that underlies the CDNs you use or are considering using will help you make better decisions.

Slap up some measurements. Look at some graphs. Make a few calls. Your site is faster. You’re a hero.

Right.

Effective Web performance is something that requires planning, preparation, execution, and the willingness to try more than once to get things right. I have discussed this problem before, but wanted to expand my thoughts into some steps that I have seen work effectively in organizations that have effectively established Web performance improvement strategies that work.

This process, in its simplest form, consists of five steps. Each step seems simple, but skipping any one of them will likely leave your Web performance process only half-baked, unable to help your team effectively improve the site.

1. Identification – What do we want/need to measure?

We want to measure everything. From everywhere.

This is an ineffective approach to Web performance measurement. This approach leads to a mass of data flowing towards you, causing your team to turn and flee, finding any way possible to hide from the coming onslaught.

Work with your team to carefully chose your Web performance targets. Identify two or three things about your site’s performance that you want to explore. Make these items discrete and clearly understood by everyone on your team. Clearly state their importance to improving Web performance. Get everyone to sign off on this.

Now, what was just said above will not be easy. There will be disagreements among people, among different parts of the organization, about which items are the most crucial to measure. This is a good thing.

Perhaps the greatest single hindrance to Web performance improvement is the lack of communication. An active debate is better than quiet acceptance and a grudging belief that you are going the wrong way. Corporate silos and a culture of assurance will not allow your company to make the decisions you need to have an effective Web performance strategy.

2. Selection – What data will we need to collect?

In order to identify a Web performance issue (which is far more important than trying to solve it), the data that will be examined will need to be decided on. This sounds easy – response time and success rate. We’re done.

Right.

Now, if your team wants to be effective, they have to understand the complexity of what they are measuring. Then an assessment of what useful data can be extract to isolate the specific performance issue under study can be made.

Choose your metrics carefully, as the wrong data is worse than no data.

3. Execution – How will we collect the data?

Once what is to be measured is decided on, the mechanics of collecting the data can be decided on. In today’s Web performance measurement environment, there are solutions to meet every preferred approach.

• Active Synthetic Monitoring. This is the old man of the methods, having been around the longest. A URL or business process is selected, scripted, and them pushed out to an existing measurement network that is managed/controlled. These have the advantage of providing static, consistent metrics that can be used as baselines for long-term trending. However, they are locked to a single process, and do not respond or indicate where your customers are going now.
• Passive User Monitoring – Browser-Side. A relative newcomer to the measurement field, this process allows companies to tag pages and follow the customer performance experience as they move through a site. This methodology can also be used to discretely measure the browser-side performance of page components that may be invisible to other measurement collection methods. It does have a weakness in that it is sometimes hard to sell within an organization because of its perceived similarity to Web analytics approaches and its need to develop an effective tagging strategy.
• Passive User Monitoring – Server-Side. This methods follows customers as they move through a site, but collects data from a users interaction with the site, rather than with the browser. Great for providing details of how customers moved through a site and how long it took to move from page to page. It is weak in providing data on how long it took for data to be delivered to the customer, and how long it took their browser to process and render the requested data.

Organizations often choose one of the methods, and stay with it. This has the effect of seeing the world through hammer goggles: If all you have is a hammer, then every problem you need to solve has to be turned into a nail.

Successful organizations have a complex, correlative approach to effective Web performance analysis. One that links performance data from multiple inputs and finds a way to link the relationships between different data sets.

If your team isn’t ready for the correlative approach, then at least keep an open mind. Not every Web performance problem is a nail.

4. Information – How do we make the data useful?

Your team now has a great lump of data, collected in a way that is understood, and providing details about things they care about.

Now what?

Web performance data is simply the raw facts that come out of the measurement systems. It is critical that during the process of determining why, what and how to measure that you also decided how you were going to process the data to produce metrics that made sense to your team.

Strategies include:

• Feeding the data into a business analytics tool
• Producing daily/weekly/monthly reports on the Key Performance Indicators (KPIs) that your team uses to measure Web performance
• Annotate change, for better or worse
• Correlate. Correlate. Correlate. Nature abhors a vacuum.

Providing a lot of raw data is the same as a vacuum – a whole bunch of nothing.

5. Action – How do we make meaningful Web performance changes?

Data has been collected and processed into meaningful data. People throughout the organization are having a-ha moments, coming up with ideas or realizations about the overall performance of the site. There are cries to just do something.

Stick to the plan. And assume that the plan will evolve in the presence of new information.

Prioritizing Web performance improvements falls into the age-old battle between the behemoths of the online business: business and IT.

Business will want to focus on issues that have the greatest effect on the bottom-line. IT will want to focus on the issues that have the greatest effect on technology.

They’re both wrong. And they’re both right.

Your online business is just that: a business that, regardless of its mission, based on technology. Effective Web performance relies on these two forces being in balance. The business cannot be successful without a sound and tuned online platform, and the technology needed to deliver the online platform cannot exist without the revenue that comes from the business done on that platform.

Effective Web performance relies on prioritizing issues so that they can be done within the business and technology plans. And an effective organization is one that has communicated (there’s that word again) what those plans are. Everyone needs to understand that the business makes decisions that effect technology and vice-versa. And that if these decisions are made in isolation, the whole organization will either implode or explode.

Takeaway

Effective Web performance is hard work. It takes a committed organization that understands that running an online business requires that everyone have access to the information they need, collected in a meaningful way, to meet the goals that everyone has agreed to.

Web performance is everywhere. People intuitively understand that when a site is slow, something’s wrong. Web performance breeds anecdotal tales of lost carts, broken catalogs, and searches gone wrong. Web performance can get you name in lights, but not in the way you or your company would like.

It’s a mistake to consider Web performance a technology problem. Web performance is really a business problem that has a technological solution.

Business problems have solutions that any mid-level executive can understand. A site that can’t handle the amount of traffic coming in requires tuning and optimization, not the firing of the current VP of Operations and a new marketing strategy.

Can you imagine the fate of the junior executive who suggested that a new marketing strategy was the solution to brick-and-mortar stores that are too small and crowded to handle the number of prospective customers (or former prospective customers) coming in the door?

Every Web performance event costs a company money, in the present and in the future. So when someone presents your company with the reality of your current Web performance, what is your response?

Some simple ideas for living with the reality that Web performance hurts business.

1. Be able to explain the issue to everyone in the company and to customers who ask. Gory details and technical mumbo-jumbo make people feel like there is something being hidden from them. Tell the truth, but make it clear what happened.
2. Do not blame anyone in public. A great way to look bad to everyone is to say that someone else caused the problem. Guess what? All that the people who visited your site during the problem will remember is that your site had the problem. Save frank discussions for behind closed doors.
3. Be able to explain to the company what the business cost was. While everyone is pointing fingers inside your company, remind them that the outage cost them $XX/minute. Of course, you can only tell them that if you know what that number is. Then gently remind everyone that this is what it cost the whole company.
4. Take real action. I don’t mean things like “We will be conducting an internal review of our processes to ensure that this is not repeated”. I mean things like listening and understanding what technology or business process failed and got you into this position in the first place. Was it someone just hitting the wrong switch? Or was it a culture of denial that did not allow the reality of Web performance to filter up to levels where real change could be implemented?
5. Demand quantitative proof that this will never happen again. Load test. Monitor. Measure. Correlate data from multiple sources. Decide how Web performance information will be communicated inside your company. Make the data available so people can ask questions. Be prepared to defend your decisions with real information.

The most successful Web companies have done thing very well. It is the core of their success and it is what makes them ruthlessly strive for Web performance excellence.

These companies understood that in order to succeed they needed to create a culture where business performance and Web performance are the same thing.

The hip new shiny thing for a new company is to position themselves as a service. Stepping back from the hype machine for a minute, can you really identify a service provider when you see one? Or are the companies that sell themselves as services are actually tools. And what differentiates a tool provider from a service provider?

A tool is designed to deliver a single unique function, such as a hammer or Twitter. Yes, Twitter is a tool. It is designed to take customer input in a variety of formats and from a number of sources and blast that content out to a variety of other formats and destinations.

Twitter is the tool. The items that feed into Twitter could be other tools (Tweetdeck as an example), or they could be true services, such as Ping.fm. What separates these two?

TweetDeck is a tool that feeds input into Twitter, and helps you manage output.

Ping.fm takes your input, and sends it where you what, modifying the format appropriately and hiding it all from you. It took the complexity of a problem (How do I post to multiple social media sites simultaneously?) and delivered a service solution, not a tool solution.

The problem with tool providers is that the problem, no matter what it is, always is a great for their tool. All customer problems fit neatly into the boundaries of what they know, and can be solved by what they sell. Tweetdeck’s answer to helping you with Twitter is to give you more Twitter your way. But it doesn’t extend or build on Twitter to create something that is truly new.

Solution providers look at the customer problem and see something new. The team at Ping.fm took a look at their personal social media management issues and found a way to create a social media input service. FriendFeed and FaceBook looked at the social media world and created a social media output service.

While tools are cool and shiny, they inevitable face the “Hammer v Screw” moment. The point when the tool reaches the outer ability of it’s ability to be useful.

Having many different hammers isn’t the solution. Heck, throwing in a wrench and a screwdriver isn’t the answer either. You’re still just selling tools.

When you step back and think about your business, when you consider what you deliver to your customers, can you really say that you deliver a service that extends and adds value to the tools you have at your disposal, that you are providing to your customers?

Or does everything just look like a nail?

I often wonder how much business is lost but the levels of assurance that exist within modern companies.

As information passes through and upward through a company, it is filtered, shaped, refined down to the one salient decision point that all the executives can then discuss. The concern that I have is whether the devolution of detail within organizations stifles their ability to innovate, especially in times of stress.

Small companies have a short distance from those that create and work with the product to the senior levels. As a result, senior managers and executives are tightly tied to the details of the product, of the company, of the customers. They understand that details are important.

Mature companies discuss how their strategies and initiatives will shape an entire industry and change the way everyone does business. But how that happens is often lost as those concepts flow downward. Just as detail devolves on the way up, detail evolves on the way down.

It is nigh on impossible to participate in an industry-defining paradigm shift when your everyday activities double and triple, leading to a complete dissociation between the executive level and the worker level.

Why does this occur?

Its not that detail devolves on the way up an organization, but rather that each level needs to assure each higher level that everything is ok and that solutions can be found for those issues that may be challenging, so lets just keep pressing forward.

So the devolution of detail coupled with the culture of assurance gets too many companies in trouble.

The devil is in the details. And sometimes, the devil can be your friend.

In the end, it’s all about performance. And when times get hard and budgets get tight, performance should be high on the list of online businesses.

The issue that I see when I work with online firms is that the first item to be on the chopping block is the performance budget. Performance appears to be equated with expense.

What name I have in the arena of Web performance that I do have I made during the last economic slowdown in 2000-2002 by writing and evangelizing some very simple facts:

1. HTTP compression is easy to set up and can reduce the bandwidth that your text files require to download. This is even more important as more and more companies build complete applications in Javascript.
2. Allow others to cache what is cacheable. Offload your site to edge and corporate servers, and let them do the heavy lifting.
3. Enable HTTP persistence. Fewer TCP connection requests make downloads run more effectively and take advantage of the benefits of the efficiencies of built into TCP.

Steve Souders’ book reminds people that designers can help this process by designing a proper page. Andy King’s book (my review) reminds people that Business and Technical Operations need to work together to reach the same goal.

So, rather than seeing this time of contraction as a challenge, take it as a chance to solidify your performance position. Retrench, re-examine, and reveal your performance issues. Integrate Web performance into the day-to-day operations of the entire business, and work within your current configuration.

The companies that learn to work more effectively and efficiently with a reduced budget and fewer people will come out on the other side as those poised to take advantage of the opportunities that are still out there.

Measure what is measurable, and make measurable what is not so.

Galileo Galilei

The greatest challenge facing companies today is not finding ways to measure performance. The key issue is one of understanding what should be measured and validating that there is agreement on what the purpose of the measurement is.

Organizations are complex. And with complexity arises the need to gather data for different purposes. In my series discussing Why Web Measurements?, I broke organizations down into four groups, each one having distinctly different needs for measurements and data. While this series focuses on Web performance, the four categories (Customer Generation, Customer Retention, Business Operations, and Technical Operations) can be broadly applied to all aspects of your business.

In each of the four categories, whether it is for Web performance or financial analysis, determining what and why to measure is a critical predecessor to the establishment of measurements and the examination of data.

2009 will be a year of reflection and retrenchment. Companies will be examining all aspects of their business, all of their relationships with vendors, all of the ways they measure themselves. The question that must be asked before succumbing to the rushing panic of cost-cutting and layoffs is: Do you fundamentally understand why and what you measure and what it is really telling you?