I will be contacting the agent locations in the next few hours to update them on this maintenance.

UPDATE: GrabPERF returned to service as of 23:40 GMT on May 30 2009. Thank you for your patience.

Tonight, I figured out how to add the Resolved IP Addresses for a host to measurement data and store that information for further debugging. It was very simple – I was trying to find complex solutions to this issue.

Turns out the solution is built right into PERL: The Socket module.

My thought is that I will update the table with the test config with three new columns:

• HTTPS/HTTP
• Hostname
• Page information

There will likely be a new table that joins with the raw data on

• Date
• Agent_id
• Test_id

And contains a comma-delimited list of all the IP addresses that the agent resolved the hostname to at the test time. This lookup will be run after the measurement, so the DNS lookup component of the measurement is not compromised.

I don’t have an ETA on this, as I want to test it fairly thoroughly before I expose the data. Adding the columns to the test config table will be transparent, but agent modification will need to be verified and then rolled out to all of the folks hosting measurement locations.

What problem does this provide a solution to?

It is vital for firms who use geographic load balancing and CDNs to verify that their data is being served from location appropriate IP addresses. I will be able to tie the information collected here into the IP-Location data I collect for other purposes and help companies ensure that this is being done.

This afternoon, the two GrabPERF Agents at Technorati were switched back to using their local copies of caching BIND for resolving DNS entries.

Some folks at Microsoft who stopped by to look at their results on the Search Performance Index noticed that there were one or two outliers in the results from these locations. When I investigated, the OpenDNS name servers I was using were returning odd results.

The reason I had been using OpenDNS is that the local BIND instances were seeing unusual behavior a few months back. So, consider the switch to the local BIND instances probational, pending ongoing review.

Thanks to Eric Schurman for letting me know.

US data shows that MSIE 7.0 is in a dominant position, with Firefox 3.0 in the 25% range of market share. This trend extends into the North American data, which is heavily influenced by the US trend.

MSIE 8.0, still reports a lower distribution than Firefox 2.0. This data is most likely based on the usage of MSIE 8.0 RC1 version, as MSIE 8.0 was only released in GA last week. It is highly probable that these stats will change in the very near future with the release of MSIE 8.0 to Windows Update.

In the EU, where fear and loathing of Microsoft runs deep and true, Firefox 3.0 is approaching parity with MSIE 7.0. Also, the perennially favoured native son Opera makes a very strong showing with their 9.6 release.

Asia is a Web designers nightmare, with MSIE 6.0 continuing to be the most reported browser. This is concerning, not simply for design reasons, but for web compliance reasons. Effectively Asia has throttled Web development to an old warhorse, but to such a degree that there must be some overriding advantage to using this browser.

As an example, the statistical comparison of four Asia nations is broken out below. We’ll start with Japan where MSIE 7.0 has a clear lead in the statistics.

However, when China (People’s Republic), India, and South Korea are added into the analysis, the pull towards MSIE 6.0 is massive.

This trend needs to be studied in greater detail in order to understand why MSIE 6.0 is so popular. Is it because of licensing? Continued use of Windows 2000? Compromised computers? The data doesn’t provide any clear or compelling reason for this trend.

Moving to Oceania shows a return to the trend of MSIE 7.0 being the predominant browser with Firefox in second place, with these two browsers showing a substantial lead over the remaining field.

South America sees MSIE 7.0 as having the largest market share, followed by MSIE 6.0 and Firefox 3.0. Effectively there are no other browsers with substantial market share at present.

These statistics show that the three most dominant browser platforms by market share are the two MSIE platforms followed by Firefox 3.0. This is likely to change with the MSIE 8.0 GA last week and its predicted release to the masses via Windows Update in the near future.

However, the release of MSIE 8.0 may not be as exponential as is predicted. Corporate IT policies, which have been slow to embrace MSIE 7.0, are likely not going to make a giant leap to MSIE 8.0 overnight. Adoption among the general population will also depend on the ability of existing Web applications to adapt to a more standards-compliant browser platform.

Noticeably absent from most of these statistics is Safari in a position to challenge the three leading browsers. This indicates that even hardcore Mac users continue to use Firefox as their primary Web application and browsing platform. StatCounter backs this up by indicating that within their data, 8.36% of visitors from the USA were on Macs, while 3.15% of visitors used Safari.

Trends to watch in the near future:

• New browser releases (Firefox 3.1, Safari 4.0) and their effect on browser distribution
• Uptake of MSIE 8.0 once it is released via Windows Update
• Browser distribution is Asia

The goal of the index is to provide performance metrics for a group of search providers around the world. The results are based on a direct HTTP GET request being made for the search results page by the GrabPERF Agent.

Currently only live data is available. In the near future, archival results on a week by week basis will be made available.

If there is a search provider that has been missed, please contact the GrabPERF team.

Today, the system was offline for several hours before I noticed that the DB had failed to restart properly. Apparently the system simply decided that the InnoDB engine had gone away.

All tables have now been switched back to the good old MyISAM and all the headaches that come with that.

The dynamic page was starting to push 20-25 seconds just for the Top 20 List. When I switched to the static list, times dropped to less than 1 second.

It’s always bad when a Web performance measurement site has poor performance.

1. The Agent code was streamlined and removed the connection error sub-routine. It seems that the latest versions of cURL no longer support the connection error determination (I can only imagine the madness of trying to support this on multiple OSs), so it has been removed as part of the error detection process. This change has been pushed out to four agents for testing and will be distributed to all other active agents after this is complete.
2. Upgrade of cURL to 7.19.4 on four agents. The same four agents that have the new agent code have also had the underlying HTTP(S) engine (cURL) upgrade to 7.19.4. Although this supports no new features that I am aware of, it is always good to be on top of the latest release with bug-fixes.

We are also trying to determine how to capture the URL that we connect to when we take a measurement. As far as cURL is documented, it still does not appear to support this feature.

There is a methodology statement explaining more about the index below the data table.

Please add any comments or questions to this post.

The switch to InnoDB was done because of the locking issues that were occurring during long queries, especially when doing ad-hoc analysis. The row-level (versus table-level) locking of InnoDB has removed most of these issues.

I have been seeing some strange behavior with the new engine. As a result of this, I will be re-starting the database engine twice a day. There should be no degradation, as this is simply a daemon re-start, not a machine re-start.