Robot Exclusion file for GrabPERF
Had to slap up a robots.txt file into the GrabPERF configuration; the MSN Crawler is getting a might aggressive. You really don’t need to index every single measurement page every single day. You don’t compete with Google by saturating the Internet with crawlers.
Advertisement
