User-Agent: ABCdatos BotLink
User-Agent: Acme.Spider 
User-Agent: Ahoy! The Homepage Finder 
User-Agent: Alkaline 
User-Agent: Anthill 
User-Agent: Walhello appie 
User-Agent: Arachnophilia 
User-Agent: Arale 
User-Agent: Araneo 
User-Agent: AraybOt 
User-Agent: ArchitextSpider 
User-Agent: Aretha 
User-Agent: ARIADNE 
User-Agent: arks 
User-Agent: ASpider (Associative Spider) 
User-Agent: ATN Worldwide 
User-Agent: Atomz.com Search Robot 
User-Agent: AURESYS 
User-Agent: BackRub 
User-Agent: unnamed 
User-Agent: BBot 
User-Agent: Big Brother 
User-Agent: Bjaaland 
User-Agent: BlackWidow 
User-Agent: Die Blinde Kuh 
User-Agent: Bloodhound 
User-Agent: Borg-Bot 
User-Agent: BoxSeaBot 
User-Agent: bright.net caching robot 
User-Agent: BSpider 
User-Agent: CACTVS Chemistry Spider 
User-Agent: Calif 
User-Agent: Cassandra 
User-Agent: Digimarc Marcspider/CGI 
User-Agent: Checkbot 
User-Agent: ChristCrawler.com 
User-Agent: churl 
User-Agent: cIeNcIaFiCcIoN.nEt 
User-Agent: CMC/0.01 
User-Agent: Collective 
User-Agent: Combine System 
User-Agent: ConfuzzledBot 
User-Agent: CoolBot 
User-Agent: Web Core / Roots 
User-Agent: XYLEME Robot 
User-Agent: Internet Cruiser Robot 
User-Agent: Cusco 
User-Agent: CyberSpyder Link Test 
User-Agent: CydralSpider 
User-Agent: Desert Realm Spider 
User-Agent: DeWeb(c) Katalog/Index 
User-Agent: DienstSpider 
User-Agent: Digger 
User-Agent: Digital Integrity Robot 
User-Agent: Direct Hit Grabber 
User-Agent: DNAbot 
User-Agent: DownLoad Express 
User-Agent: DragonBot 
User-Agent: DWCP (Dridus' Web Cataloging Project) 
User-Agent: e-collector 
User-Agent: EbiNess 
User-Agent: EIT Link Verifier Robot 
User-Agent: ELFINBOT 
User-Agent: Emacs-w3 Search Engine 
User-Agent: ananzi 
User-Agent: esculapio 
User-Agent: Esther 
User-Agent: Evliya Celebi 
User-Agent: nzexplorer 
User-Agent: FastCrawler 
User-Agent: Fluid Dynamics Search Engine robot 
User-Agent: Felix IDE 
User-Agent: Wild Ferret Web Hopper #1, #2, #3 
User-Agent: FetchRover 
User-Agent: fido 
User-Agent: Hämähäkki 
User-Agent: KIT-Fireball 
User-Agent: Fish search 
User-Agent: Fouineur 
User-Agent: Robot Francoroute 
User-Agent: Freecrawl 
User-Agent: FunnelWeb 
User-Agent: gammaSpider, FocusedCrawler 
User-Agent: gazz 
User-Agent: GCreep 
User-Agent: GetBot 
User-Agent: GetURL 
User-Agent: Golem 
User-Agent: Googlebot 
User-Agent: Grapnel/0.01 Experiment 
User-Agent: Griffon 
User-Agent: Gromit 
User-Agent: Northern Light Gulliver 
User-Agent: Gulper Bot 
User-Agent: HamBot 
User-Agent: Harvest 
User-Agent: havIndex 
User-Agent: HI (HTML Index) Search 
User-Agent: Hometown Spider Pro 
User-Agent: Wired Digital 
User-Agent: ht://Dig 
User-Agent: HTMLgobble 
User-Agent: Hyper-Decontextualizer 
User-Agent: iajaBot 
User-Agent: IBM_Planetwide 
User-Agent: Popular Iconoclast 
User-Agent: Ingrid 
User-Agent: Imagelock 
User-Agent: IncyWincy 
User-Agent: Informant 
User-Agent: InfoSeek Robot 1.0 
User-Agent: Infoseek Sidewinder 
User-Agent: InfoSpiders 
User-Agent: Inspector Web 
User-Agent: IntelliAgent 
User-Agent: I, Robot 
User-Agent: Iron33 
User-Agent: Israeli-search 
User-Agent: JavaBee 
User-Agent: JBot Java Web Robot 
User-Agent: JCrawler 
User-Agent: AskJeeves 
User-Agent: JoBo Java Web Robot 
User-Agent: Jobot 
User-Agent: JoeBot 
User-Agent: The Jubii Indexing Robot 
User-Agent: JumpStation 
User-Agent: image.kapsi.net 
User-Agent: Katipo 
User-Agent: KDD-Explorer 
User-Agent: Kilroy 
User-Agent: KO_Yappo_Robot 
User-Agent: LabelGrabber 
User-Agent: larbin 
User-Agent: legs 
User-Agent: Link Validator 
User-Agent: LinkScan 
User-Agent: LinkWalker 
User-Agent: Lockon 
User-Agent: logo.gif Crawler 
User-Agent: Lycos 
User-Agent: Mac WWWWorm 
User-Agent: Magpie 
User-Agent: marvin/infoseek 
User-Agent: Mattie 
User-Agent: MediaFox 
User-Agent: MerzScope 
User-Agent: NEC-MeshExplorer 
User-Agent: MindCrawler 
User-Agent: mnoGoSearch search engine software 
User-Agent: moget 
User-Agent: MOMspider 
User-Agent: Monster 
User-Agent: Motor 
User-Agent: MSNBot 
User-Agent: Muncher 
User-Agent: Muninn 
User-Agent: Muscat Ferret 
User-Agent: Mwd.Search 
User-Agent: Internet Shinchakubin 
User-Agent: NDSpider 
User-Agent: NetCarta WebMap Engine 
User-Agent: NetMechanic 
User-Agent: NetScoop 
User-Agent: newscan-online 
User-Agent: NHSE Web Forager 
User-Agent: Nomad 
User-Agent: The NorthStar Robot 
User-Agent: ObjectsSearch 
User-Agent: Occam 
User-Agent: HKU WWW Octopus 
User-Agent: OntoSpider 
User-Agent: Openfind data gatherer 
User-Agent: Orb Search 
User-Agent: Pack Rat 
User-Agent: PageBoy 
User-Agent: ParaSite 
User-Agent: Patric 
User-Agent: pegasus 
User-Agent: The Peregrinator 
User-Agent: PerlCrawler 1.0 
User-Agent: Phantom 
User-Agent: PhpDig 
User-Agent: PiltdownMan 
User-Agent: Pimptrain.com's robot 
User-Agent: Pioneer 
User-Agent: html_analyzer 
User-Agent: Portal Juice Spider 
User-Agent: PGP Key Agent 
User-Agent: PlumtreeWebAccessor 
User-Agent: Poppi 
User-Agent: PortalB Spider 
User-Agent: psbot 
User-Agent: GetterroboPlus Puu 
User-Agent: The Python Robot 
User-Agent: Raven Search 
User-Agent: RBSE Spider 
User-Agent: Resume Robot 
User-Agent: RoadHouse Crawling System 
User-Agent: RixBot 
User-Agent: Road Runner: The ImageScape Robot 
User-Agent: Robbie the Robot 
User-Agent: ComputingSite Robi/1.0 
User-Agent: RoboCrawl Spider 
User-Agent: RoboFox 
User-Agent: Robozilla 
User-Agent: Roverbot 
User-Agent: RuLeS 
User-Agent: SafetyNet Robot 
User-Agent: Scooter 
User-Agent: Search.Aus-AU.COM 
User-Agent: Sleek 
User-Agent: SearchProcess 
User-Agent: Senrigan 
User-Agent: SG-Scout 
User-Agent: ShagSeeker 
User-Agent: Shai'Hulud 
User-Agent: Sift 
User-Agent: Simmany Robot Ver1.0 
User-Agent: Site Valet 
User-Agent: SiteTech-Rover 
User-Agent: Skymob.com 
User-Agent: SLCrawler 
User-Agent: Inktomi Slurp 
User-Agent: Smart Spider 
User-Agent: Snooper 
User-Agent: Solbot 
User-Agent: Speedy Spider 
User-Agent: spider_monkey 
User-Agent: SpiderBot 
User-Agent: Spiderline Crawler 
User-Agent: SpiderMan 
User-Agent: SpiderView(tm) 
User-Agent: Spry Wizard Robot 
User-Agent: Site Searcher 
User-Agent: Suke 
User-Agent: suntek search engine 
User-Agent: Sven 
User-Agent: Sygol 
User-Agent: TACH Black Widow 
User-Agent: Tarantula 
User-Agent: tarspider 
User-Agent: Tcl W3 Robot 
User-Agent: TechBOT 
User-Agent: Templeton 
User-Agent: TitIn 
User-Agent: TITAN 
User-Agent: The TkWWW Robot 
User-Agent: TLSpider 
User-Agent: UCSD Crawl 
User-Agent: UdmSearch 
User-Agent: UptimeBot 
User-Agent: URL Check 
User-Agent: URL Spider Pro 
User-Agent: Valkyrie 
User-Agent: verticrawl 
User-Agent: Victoria 
User-Agent: vision-search 
User-Agent: void-bot 
User-Agent: Voyager 
User-Agent: VWbot 
User-Agent: The NWI Robot 
User-Agent: W3M2 
User-Agent: WallPaper (alias crawlpaper) 
User-Agent: the World Wide Web Wanderer 
User-Agent: w@pSpider by wap4.com 
User-Agent: WebBandit Web Spider 
User-Agent: WebCatcher 
User-Agent: WebCopy 
User-Agent: webfetcher 
User-Agent: The Webfoot Robot 
User-Agent: Webinator 
User-Agent: weblayers 
User-Agent: WebLinker 
User-Agent: WebMirror 
User-Agent: The Web Moose 
User-Agent: WebQuest 
User-Agent: Digimarc MarcSpider 
User-Agent: WebReaper 
User-Agent: webs 
User-Agent: Websnarf 
User-Agent: WebSpider 
User-Agent: WebVac 
User-Agent: webwalk 
User-Agent: WebWalker 
User-Agent: WebWatch 
User-Agent: Wget 
User-Agent: whatUseek Winona 
User-Agent: WhoWhere Robot 
User-Agent: Weblog Monitor 
User-Agent: w3mir 
User-Agent: WebStolperer 
User-Agent: The Web Wombat 
User-Agent: The World Wide Web Worm 
User-Agent: WWWC Ver 0.2.5 
User-Agent: WebZinger 
User-Agent: XGET 
User-Agent: Nederland.zoek 

Disallow: /cgi-bin/ 
Disallow: /tmp/ 
Disallow: /_private/