User-agent: * Disallow: /indeximages User-agent: * Disallow: /recipbnr User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; Dow Jones Searchbot) Disallow: / User-agent: SheenBot/SheenBot-1.0.2 (Sheen web crawler) Disallow: / User-agent: Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) Disallow: / User-agent: taptubot *** please read http://www.taptu.com/corp/taptubot *** Disallow: / User-agent: Comodo-Certificates-Spider Disallow: / User-agent: Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/) Disallow: / User-agent: Mozilla/5.0 (compatible; LegalAnalysisAgent/1.0; http://www.legalx.net) Disallow: / User-agent: Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/) Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools) Disallow: / User-agent: psbot/0.1 (+http://www.picsearch.com/bot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) Disallow: / User-agent: Mozilla/4.0 (compatible;++see+http://www.parchmenthill.com/search.htm) Disallow: / User-agent: Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm) Disallow: / User-agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620 Disallow: / User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/) Disallow: / User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com) Disallow: / User-agent: Jakarta Commons-HttpClient/3.0 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: panscient.com Disallow: / User-agent: AboutUsBot Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: WikiBot Disallow: / User-agent: WikipediaBot Disallow: / User-agent: panscient.com Disallow: / User-agent: MSIECrawler Disallow: / User-agent: Twiceler Disallow: / User-agent: Asterias Disallow: / User-agent: BotRightHere Disallow: / User-agent: BlowFish Disallow: / User-agent: BotALot Disallow: / User-agent: Bookmark Disallow: / User-agent: Bullseye Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: BackDoorBot Disallow: / User-agent: Copernic Disallow: / User-agent: CherryPicker Disallow: / User-agent: Crescent Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPickerSE Disallow: / User-agent: CherryPickerElite Disallow: / User-agent: Cosmos Disallow: / User-agent: DittoSpyder Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: EStripper Disallow: / User-agent: EmailStripper Disallow: / User-agent: EmailStrip Disallow: / User-agent: FlamingAttackBot Disallow: / User-agent: GetRight Disallow: / User-agent: Gaisbot Disallow: / User-agent: Harvest Disallow: / User-agent: HarvestBot Disallow: / User-agent: Hloader Disallow: / User-agent: Humanlinks Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LexiBot Disallow: / User-agent: LinkScan Disallow: / User-agent: LinkStripper Disallow: / User-agent: LinkStrip Disallow: / User-agent: MIIxpc Disallow: / User-agent: MisterPiX Disallow: / User-agent: NetMechanic Disallow: / User-agent: NICErsPRO Disallow: / User-agent: NetAnts Disallow: / User-agent: Openfind Disallow: / User-agent: Openbot Disallow: / User-agent: Python Disallow: / User-agent: ProWebWalker Disallow: / User-agent: ProPowerBot Disallow: / User-agent: RMA Disallow: / User-agent: RadiationRetriever Disallow: / User-agent: Stripper Disallow: / User-agent: Strippa Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Spanner Disallow: / User-agent: Scanner Disallow: / User-agent: SpankBot Disallow: / User-agent: SpiderWeb Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Telesoft Disallow: / User-agent: TheNomad Disallow: / User-agent: True_Robot Disallow: / User-agent: TrueRobot Disallow: / User-agent: TightTwatBot Disallow: / User-agent: ToCrawl Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: URLBot Disallow: / User-agent: URLCrawler Disallow: / User-agent: WebZip Disallow: / User-agent: WebsiteQuester Disallow: / User-agent: WebStripper Disallow: / User-agent: WebSauger Disallow: / User-agent: WebCopier Disallow: / User-agent: WebBandit Disallow: / User-agent: Wget Disallow: / User-agent: WebImageCollector Disallow: / User-agent: WebsterPro Disallow: / User-agent: WebBandit Disallow: / User-agent: XenusLinkSleuth Disallow: / User-agent: Xenus Disallow: / User-agent: Zeus Disallow: / User-agent: ZeusLinkScout Disallow: /