User-agent: Googlebot Allow: / User-agent: msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) #url: http://search.msn.com/msnbot.htm #note: msnbots have changed their name recently Crawl-delay: 5 User-agent: MSNBot #url: http://www.msn.com/ #note: msn also accepts a delay Crawl-delay: 5 User-agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm) #url: http://search.msn.com/msnbot.htm #note: msnbots have changed their name recently Crawl-delay: 5 User-agent: Slurp #url: http://help.yahoo.com/help/us/ysearch/slurp/index.html #note: inktomi's "slurp" can read a minimum delay between hits Crawl-delay: 5 User-agent: Yahoo! Slurp Crawl-delay: 5 User-agent: RyteBot Crawl-delay: 5 User-agent: LRTBot Crawl-delay: 5 User-agent: daumoa Disallow: / User-agent: Yanga WorldSearch Bot Disallow: / User-agent: jobo Disallow: / User-agent: EchO! Disallow: / User-agent: LinkChecker Disallow: / User-agent: Fish search Disallow: / User-agent: yacybot Disallow: / User-agent: Bloglines Disallow: / User-agent: Twiceler Disallow: / User-agent: Zealbot Disallow: / User-agent: VoilaBot Disallow: / User-agent: appie Disallow: / User-agent: sun4u Disallow: / User-agent: HTTrack Disallow: / User-agent: Nutch Disallow: / User-agent: Sisi Disallow: / User-agent: FyberSpider Disallow: / User-agent: girafabot Disallow: / User-agent: lmspider Disallow: / User-agent: NP Disallow: / User-agent: Robi Disallow: / User-agent: Webster Pro Disallow: / User-agent: Webster Disallow: / User-agent: Zeus Disallow: / User-agent: Gigabot Disallow: / User-agent: Scirus Disallow: / User-agent: Speedy Spider Disallow: / User-agent: PicoSearch Disallow: / User-agent: Plucker Disallow: / User-agent: DISCo Pump Disallow: / User-agent: Gulliver Disallow: / User-agent: vspider Disallow: / User-agent: EmailSiphon Disallow: / User-agent: Teleport Pro Disallow: / User-agent: Fetch Disallow: / User-agent: pamuk Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCapture Disallow: / User-agent: Mass Downloader Disallow: / User-agent: WebCopy Disallow: / User-agent: AWV0.8d Disallow: / User-agent: Crescent Internet ToolPak Disallow: / User-agent: JOC Web Spider Disallow: / User-agent: WebStripper Disallow: / User-agent: SiteSucker Disallow: / User-agent: Webdup Disallow: / User-agent: Scooter Disallow: / User-agent: Python-urllib Disallow: / User-agent: Python Disallow: / User-agent: Franklin Locator Disallow: / User-agent: CK-SillyDog Disallow: / User-agent: PocketHTTP Disallow: / User-agent: VoilaBot Disallow: / User-agent: Xaldon WebSpider Disallow: / User-agent: WebCapture Disallow: / User-agent: WebStripper Disallow: / User-agent: Java Disallow: / User-agent: WebReaper Disallow: / User-agent: TeragramWebcrawler Disallow: / User-agent: Vagabondo Disallow: / User-agent: nogoop-HttpClient Disallow: / User-agent: Baiduspider Disallow: / User-agent: MyOperaTB/1.0 Disallow: / User-agent: MyOperaTB Disallow: /