Newest Industry
Web Performance, Branding, and Social Media
Cool features of the Slurp! Bot
In: smp
11 Feb 2005Yahoo Search Blog has a great posting on how the Slurp Bot tries to conserve bandwidth by making use of compression and cache-control headers. [here]
As a Web performance fanatic, it is heartening to see that these folks have taken such care, and put such thought into their indexing crawler. They want it to be accurate, but they don’t want to slam your site.
A while back, I had to write a robots.txt file for WebPerformance to keep the MSNBot from stomping the site on a daily basis. This site uses frames and query variables to produce the various performance graphs. Well, the MSNBot was indexing every page and every variation almost daily. Finally, I said go away, just to that crawler. All the others are fine. Maybe MSN Search should take a page from the Yahoo! (Inktomi) Bot development team.
Related Posts
- Tags: bandwidth, blog, cache-control, compress, compression, control, cool, EAD, HTML, HTTP, IE, it, Om, performance, posting, pr, search, the other, variation, web, Web Performance, WebPerformance.Org, Yahoo!
-
design4speed