达维营-前端网| Crawler | User agent | Owner |
|---|---|---|
| 192.com | 192.comAgent | 192.com |
| 4anything | 4anything.com LinkChecker v2.0 | 4anything |
| A-Online | A-Online Search | Aon |
| ABCdatos | ABCdatos BotLink/5.xx.xxx#BBL | ABCdatos |
| AOL | Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) | AOL |
| ASAHA | ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) | ASAHA |
| ASPseek | ASPSeek/1.2.xa | ASPseek |
| ASPSeek/1.2.xxpre | ||
| ASPSeek/1.2.x | ||
| ASPSeek/1.2.5 | ||
| ASPseek/1.2.9d | ||
| ASPseek/1.2.xx | ||
| AVSearch | AVSearch-1.0(peter.turney@nrc.ca) | National Research Council Canada |
| AbachoBot | AbachoBOT | Abacho |
| AbachoBOT (Mozilla compatible) | ||
| Aberja Checkoma | Aberja Checkoma | Aberja |
| Abot | abot/0.1 (abot; http://www.abot.com; abot@abot.com) | Abot.com |
| About | About/0.1libwww-perl/5.47 | About |
| AboutUsBot | Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot) | AboutUs |
| Accelobot | Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com) | Accelovation |
| Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.accelobot.com) | ||
| Accoona | Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) | Accoona |
| accoona | ||
| Acoi | AcoiRobot | Acoi |
| Acoon Robot | Acoon Robot v1.50.001 | Acoon |
| Acoon-Robot | Acoon-Robot v3.00 (http://www.acoon.de and http://www.acoon.com) | Acoon |
| Acorn | Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot o rg) |
Isara |
| Activtourist | Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) | Activtourist |
| Aesop | AESOP_com_SpiderMan | Aesop |
| Agada | Mozilla/4.0 (agadine3.0) www.agada.de | Agada |
| agadine/1.x.x (+http://www.agada.de) | ||
| Mozilla/4.0 (agadine3.0) | ||
| AgentName | AgentName/0.1 libwww-perl/5.48 | Linkomatic |
| Aibot | AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , C hina) |
21seek |
| Aicrawler | Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) | Accoona |
| Aipbot | aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) | Aipbot |
| Alacra | PortalBSpider/2.0 (spider@portalb.com) | Alacra |
| Aladin.de | Aladin/3.324 | Abacho |
| Aleksika Danmark | Aleksika Spider/1.0 (+http://www.aleksika.com/) | Aleksika |
| Alexa | ia_archiver | Alexa |
| AlkalineBOT | AlkalineBOT/1.4 (1.4.0326.0 RTM) | Vestris |
| AlkalineBOT/1.3 | ||
| Allesklar.de | Allesklar/0.1 libwww-perl/5.46 | Allesklar |
| Allrati | Allrati/1.1 (+) | Unknown |
| Almaden | http://www.almaden.ibm.com/cs/crawler [hc5] | IBM |
| http://www.almaden.ibm.com/cs/crawler | ||
| Altavista | Scooter-W3-1.0 | Altavista |
| Scooter2_Mercator_x-x.0 | ||
| scooter-venus-3.0.vns | ||
| Scooter-W3.1.2 | ||
| Scooter-3.2.NIV | ||
| Scooter-3.2.JT | ||
| Scooter-3.2.EX | ||
| Scooter/2.0 G.R.A.B V1.0 | ||
| Scooter_trk3-3.0.3 | ||
| Scooter-3.2.SF0 | ||
| Scooter/3.3.QA.pczukor | ||
| Scooter/3.3 | ||
| Scooter/1.1 (custom) | ||
| Scooter-3.2.DIL | ||
| Scooter/1.0 scooter@pa.dec.com | ||
| Scooter-3.0.FS | ||
| Scooter-3.0.EU | ||
| Scooter/2.0 G.R.A.B. X2.0 | ||
| Scooter/2.0 G.R.A.B. V1.1.0 | ||
| Scooter-3.0.HD | ||
| Scooter-3.0.VNS | ||
| Scooter-3.2.BT | ||
| Scooter-3.2 | ||
| Scooter-3.0QI | ||
| Scooter/3.3_SF | ||
| Scooter-3.2.snippet | ||
| Scooter-ARS-1.1-ih | ||
| Scooter/3.3.vscooter | ||
| Scooter-3.3dev | ||
| Scooter-ARS-1.1 | ||
| Scooter_bh0-3.0.3 | ||
| Scooter/1.0 | ||
| Amfibibot | Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) | Amfibi |
| Amidalla | libwww-perl/5.65 | Amidalla |
| Annomille | AnnoMille spider 0.1 alpha – http://www.annomille.it | Annomille |
| AnsearchBot | Mozilla/5.0 (compatible; AnsearchBot/1.0; +http://www.ansearch.com.au/) | Ansearch |
| AnswerBus | AnswerBus (http://www.answerbus.com/) | AnswerBus |
| Answerchase | PROve AnswerBot 4.0 | Answerchase |
| Antibot | antibot-V1.3.3.1/debian-linux-sarge | Antidot |
| Any Search Info | Mozilla/4.0 (Sleek Spider/1.2) | Search-Info |
| Anzwers Australia | AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) | Anzwers Australia |
| Apexoo Spider | Apexoo Spider 1.0 | Apexoo |
| Aport | Aport | Aport |
| Appie | appie 1.1 (www.walhello.com) | Walhello |
| appie 1.1 (www.walhello.com) | ||
| Arabulbot | Mozilla/5.0 (compatible; arabulbot/1.1; +http://www.arabul.com/bot.html) | Arabul |
| ArabyBot | ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby .com;) |
Araby |
| Arachnoidea | Arachnoidea (arachnoidea@euroseek.com) | Euroseek |
| ArchitextSpider | ArchitextSpider | Excite |
| Archive.org_bot | Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +ht tp://www.archive-it.org) |
Archive.org |
| Arexera | TECOMAC-Crawler/0.x | Arexera |
| X-Crawler | ||
| Arianna | www.arianna.it | Libero |
| Arikus_Spider | Arikus_Spider | Arikus |
| Asahina | Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) | Asahina |
| Asahina-Antenna/1.x | ||
| Ask 24x Info | ask.24x.info | Ask 24x |
| Ask Jeeves/Teoma | Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w ebmasters.shtml) |
Ask Jeeves |
| Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/w ebmasters.shtml) |
||
| Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/we bmasters.shtml) |
||
| Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_cr awling.html) |
||
| Asked | asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) | Asked |
| Askpeter_bot | Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info) | Askpeter |
| Asterias | asterias/2.0 | Singing Fish |
| Asterias Crawler | Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revis ion: 3.11 |
Singingfish |
| Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.s ingingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revisi on: 3.10 |
||
| Astrafind! | Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) | Seeq |
| Atlocal | AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) | @Local |
| Attentio | Attentio/Nutch-0.9-dev (Attentio’s beta blog crawler; www.attentio.com; info@att entio.com) |
Attentio |
| Augurnet Swiss | augurfind | Augurnet Swiss |
| augurnfind V-1.x | ||
| Axada | axadine/ (Axadine Crawler; http://www.axada.de/; ) | Axada |
| Axandra | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) | Axandra |
| Axmo | AxmoRobot – Crawling your site for better indexing on www.axmo.com search engine . |
Axmo |
| Ay-Up | FastBug http://www.ay-up.com | Ay-up |
| BE Internet Search Engine | Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) | Rawgrunt |
| Ba.be | Mozilla/4.72 [en] (BACS http://www.ba.be) | BA |
| BaBoom Web Portal | BaboomBot/1.x.x (+http://www.baboom.us) | Baboum |
| BabalooSpider | BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) | Babaloo |
| Backlink-Check | Backlink-Check.de (+http://www.backlink-check.de/bot.html) | Backlink-Check |
| Baiduspider | Baiduspider+(+http://www.baidu.com/search/spider.htm) | Baidu.com |
| Baiduspider+(+http://www.baidu.com/search/spider_jp.html) | ||
| Balihoo | Bloodhound/Nutch-0.9 (Testing Crawler for Research – obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) |
Balihoo |
| TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.a spx; tgautier at balihoo dot com) |
||
| BanBot | BanBots/1.2 (spider@banbots.com) | Banbot |
| BeamMachine | BeamMachine/0.5 (dead link remover of www.beammachine.net) | BeamMachine |
| Beauty (Cosmoty) | beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) | uCHOOSE |
| BebopBot | BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) | Apassion4jazz |
| BecomeBot | Mozilla/5.0 (compatible; BecomeBot/1.83; MSIE 6.0 compatible; +http://www.become .com/site_owners.html) |
BecomeBot |
| Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become. com/site_owners.html) |
||
| BecomeJPBot | Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.becom e.co.jp/site_owners.html) |
Become |
| BeijingCrawler | BeijingCrawler | Unknown |
| BigClique | BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) | BigClique |
| Biglotron | BIGLOTRON (Beta 2;GNU/Linux) | Biglotron |
| Bigsearch | Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c a/; info@enhancededge.com) |
Bigsearch |
| Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.c a/; info@enhancededge.com) |
||
| BilBasen | FAST Enterprise Crawler 6 used by BilBasen ApS (michael@bilinfo.dk) | Bilinfo |
| BilgiBetaBot | BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html ; nutch-agent@lucene.apache.org) |
Bilgi |
| BilgiBot | BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) | Bilgi |
| Bisnisseek | Custom Spider www.bisnisseek.com /1.0 | Bisnisseek |
| Bitacle Robot | Bitacle bot/1.1 | Bitacle |
| Bitacle Robot (V:1.0;) (http://www.bitacle.com) | ||
| Blaiz Enterprises | Blaiz-Bee/1.0 (+http://www.blaiz.net) | Blaiz Enterprises |
| Blaiz-Bee | Blaiz-Bee/2.00.5502 (+http://www.blaiz.net) | Blaiz |
| Blaiz-Bee/2.00.5622 ( http://www.blaiz.net) | ||
| Blitzsuche | BlitzBOT@tricus.net | RP ONLINE |
| Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T) | ||
| BlitzBOT@tricus.net (Mozilla compatible) | ||
| BlogRefsBot | Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers ) |
BlogRefs |
| BlogSearch | BlogSearch/1.0 +http://www.icerocket.com/ | IceRocket |
| BlogSearch/1.x +http://www.icerocket.com/ | ||
| BlogWatcher | blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) | Okumura Group |
| Blogbot | Naamah 1.0a/Blogbot (http://blogbot.de/) | Blogbot |
| Naamah 1.0.1/Blogbot (http://blogbot.de/) | ||
| Blogdex | BlogBot/1.x | Massachusetts Institute of Technology |
| Blogdimension BlogBot | Blogdimension/Alpha2 (Blogdimension BlogBot; http://www.blogdimension.com) | Blogdimension |
| Bloglines | Bloglines Title Fetch/1.0 (http://www.bloglines.com) | Bloglines |
| Bloglines/3.1 (http://www.bloglines.com) | ||
| Bloglines-Images | Bloglines-Images/0.1 (http://www.bloglines.com) | Bloglines |
| BlogzIce | BlogzIce/1.0 +http://www.icerocket.com/ | IceRocket |
| BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com) | ||
| Boitho | boitho.com-robot/1.x (http://www.boitho.com/bot.html) | Boitho |
| boitho.com-dc | ||
| boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html) | ||
| boitho.com-robot/1.x | ||
| Bot | bot/1.0 (bot; http://; bot@bot.bot) | Unknown |
| BotSeer | Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) | Penn State College of Information Sciences and Technology |
| Botmobi | Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://fi nd.mobi/bot.html abuse@mtld.mobi) |
Find.mobi |
| BravoBrian bSTOP | BStop.BravoBrian.it Agent Detector | BravoBrian |
| BravoBrian SpiderEngine MarcoPolo | ||
| BrightCrawler | BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) | Brightcloud |
| Bruinbot | BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) | University of California |
| Btbot | BTbot/0.x (+http://www.btbot.com/btbot.html) | Btbot |
| BuildCMS crawler | BuildCMS crawler (http://www.buildcms.com/crawler) | BuildCMS |
| BuiltWith | Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html) | BuiltWith |
| BullsEye/Intelliseek | BullsEye | Intelliseek |
| BurstFindCrawler | BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawl er@burstfind.com) |
Burstfind |
| Buscaplus | Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) | Buscaplus |
| CEA | larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) | CEA |
| CMP | libwww-perl/5.41 | CMP United Business Media |
| CUPS | PrivacyFinder Cache Bot v1.0 | PrivacyBird |
| Camcrawler | Camcrawler (+http://www.camdiscover.com/crawler.html) | Sensation Internet Services |
| CanadianContent Search | RoboCrawl (http://www.canadiancontent.net) | CanadianContent |
| RoboCrawl (www.canadiancontent.net) | ||
| Carleson | carleson/1.0 | Cosmix |
| Catall Spider | Catall Spider | Catall |
| Catall-Spider | Catall-Spider/3.3.3(www.Catall.de) | Catall |
| CazoodleBot | CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.co m) |
Cazoodle |
| CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazood lebot; cazoodlebot@cazoodle.com) |
||
| Ccubee | ccubee/x.0 | Empyreum |
| Changedetection | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetect ion.com/bot.html ) |
Changedetection |
| Charlotte | Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) | Searchme |
| Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/) | ||
| Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com) | ||
| Christcentral | Mozilla/4.0 (compatible; ChristCrawler.com, ChristCrawler@ChristCENTRAL.com) | Christcentral |
| ChristCRAWLER 2.0 | ||
| CipinetBot | CipinetBot (http://www.cipinet.com/bot.html) | Cipinet |
| CipinetBot/1.0 (http://www.cipinet.com/bot.html) | ||
| CjLogbot | Mozilla/5.0 (compatible; CjLogbot 1.0; +http://www.cjlog.com/bot) | CjLog |
| Claymont Search | Claymont.com | Claymont Search |
| CloakDetect | CloakDetect/0.9 (+http://fulltext.seznam.cz/) | Seznam |
| Clushbot | Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html) | Clush |
| Clushbot/2.x (+http://www.clush.com/bot.html) | ||
| Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/) | ||
| Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html) | ||
| Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html) | ||
| Clushbot/3.31-BinaryFury (+http://www.clush.com/bot.html) | ||
| Clushbot/3.xx-Hector (+http://www.clush.com/bot.html) | ||
| Cnet robot | Mozilla/4.6 [en] (http://www.cnet.com/) | Search.com |
| CoBITSProbe | CoBITSProbe | Academia Sinica |
| Cobion | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) | Cobion |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) | ||
| oBot ((compatible;Win32)) | ||
| Combine | Combine/2.0 http://combine.it.lth.se/ | Combine |
| Combine/2.0 | ||
| Combine/3 http://combine.it.lth.se/ | ||
| Cometrics-bot | cometrics-bot, http://www.cometrics.de | Cometrics |
| Cometsystems | Crawler (cometsearch@cometsystems.com) | Cometsystems |
| Crawler (cometsearch@cometsystems.com) | ||
| Comperio | FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) | Comperio |
| Compete.com | larbin_2.2.0 (crawl@compete.com) | Compete Inc |
| Computerorgs | htdig/3.1.6 (http://computerorgs.com) | Computerorgs.com |
| Comrite | Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucen e.apache.org) |
Comrite |
| ConveraCrawler | ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl) | Convera |
| ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) | ||
| Converas RetrievalWare | CrawlConvera0.1 (CrawlConvera@yahoo.com) | Convera |
| ConveraCrawler/0.2 | ||
| Convera Internet Spider V6.x | ||
| ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) | ||
| CoolBot | CoolBot | SuchMaschine21 |
| Cortina | Vision Research Lab image spider at vision.ece.ucsb.edu | Vision Research Lab |
| CougarSearch | CougarSearch/0.1 (+http://www.cougarsearch.com/faq.shtml) | CougarSearch |
| Cowbot | Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) | Naver |
| Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) | ||
| CrawlerBoy | CrawlerBoy Pinpoint.com | Motricity |
| Crawling jpeg | Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) | Yamana Laboratory – Waseda University Japan |
| Crawllybot | Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) | Crawlly |
| Croccrawler | CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) | Croccrawler.com |
| CsCrawler | Hi! I’m CsCrawler, my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/goo glespam/crawler.html RPT-HTTPClient/0.3-3 |
University of Kassel |
| Csci_b659/0.13 | csci_b659/0.13 | Indiana University School of Informatics |
| Cuasar | Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ | Cuasar |
| CurryGuide | CurryGuide SiteScan 1.1 | CurryGuide |
| CyberAlerts | Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) | CyberAlerts |
| Cydral | CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) | Cydral |
| CydralSpider | CydralSpider/2.4 (Cydral Image Search; http://www.cydral.com) | Cydral |
| CydralSpider/2.2 (Cydral Image Search; http://www.cydral.com) | ||
| DAUM RSS Robot | ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net /aboutkr.html) |
Daum |
| DAUM Web Robot | Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.0; DAUM Web Robot; Daum Comm unications Corp., Korea) |
Daum |
| Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Comm unications Corp., Korea) |
||
| Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Commu nications Corp., Korea; +http://ws.daum.net/aboutkr.html) |
||
| Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communi cations Corp., Korea; +http://ws.daum.net/aboutkr.html) |
||
| DNS-Digger | Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) | Dnsdigger |
| DailyOrbit | Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) | DailyOrbit |
| DataFountains | DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents .shtml) |
University of California |
| DataFountains/DMOZ Downloader | ||
| DataSpear Spider Bot | DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.co m) |
DataSpear |
| DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html ; dssb@dataspear.com) |
||
| DataparkSearch | DataparkSearch/4.xx (http://www.dataparksearch.org/) | dpSearch |
| DataparkSearch/4.47 (+http://dataparksearch.org/bot) | ||
| DaviesBot | DaviesBot/1.7 (www.wholeweb.net) | Wholeweb |
| Daypop | daypopbot/0.x | Daypop |
| DbDig | dbDig(http://www.prairielandconsulting.com) | Connections |
| De.com | Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) | De.com |
| DeepIndexer | DeepIndexer.ca | Deepindex |
| Deepak-USC/ISI | deepak-USC/ISI | University of Southern California |
| Deepindex | DeepIndex (www.en.deepindex.com) | Deepindex |
| DeepIndex | ||
| Deepindex V2 | ||
| Denmex Websearch | Denmex websearch (http://search.denmex.com) | Denmex Websearch |
| DepSpid | Mozilla/4.0 (compatible; DepSpid/5.03; +http://about.depspid.net) | DepSpid |
| Dev-spider2 | dev-spider2.searchpsider.com/1.3b | Searchspider |
| DiaGem Japan | DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) | DiaGem Japan |
| Die Kraehe | -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de | Die Kraehe |
| Diggit | Digger/1.0 JDK/1.3.0rc3 | Diggit |
| Direct Hit | Mozilla/2.0 (compatible; EZResult — Internet Search Engine) | Teoma |
| Disco-crawl | disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@ discoveryengine.com) |
Discoveryengine |
| disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@disc overyengine.com) |
||
| Ditto | DittoSpyder | Ditto |
| DoCoMo | DoCoMo/1.0/Nxxxi/c10 | NTT DoCoMo |
| DoCoMo/1.0/Nxxxi/c10/TB | ||
| DoCoMo/2.0 P900iV(c100;TB;W24H11) | ||
| Dodgebot | dodgebot/experimental | Agmlab |
| DotBot | Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotne tdotcom.org) |
Dotnetdotcom |
| DotBot/1.0.1 | ||
| Doubanbot | Doubanbot/1.0 (bot@douban.com http://www.douban.com) | Douban |
| Download-Tipp | Download-Tipp Linkcheck (http://download-tipp.de/) | Download-Tipp |
| EyeCatcher (Download-tipp.de)/1.0 | ||
| Drecombot | Drecombot/1.0 (http://career.drecom.jp/bot.html) | Drecom Japan |
| DtSearchSpider | dtSearchSpider | dtSearch |
| Dumbot | Dumbot(version 0.1 beta) | DumbFind.com |
| Dumbot(version 0.1 beta – http://www.dumbfind.com/dumbot.html) | ||
| Dumbot(version 0.1 beta – dumbfind.com) | ||
| E-SocietyRobot | e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) | Yamana Laboratory |
| E-StyleISP | eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) | e-StyleISP |
| EApolloBot | eApolloBot/1.0 (eApollo search engine robot; http://www.eapollo.com; eapollo at global-opto dot com) |
EApollo |
| EMPAS_ROBOT | EMPAS_ROBOT | Empas |
| ESISmartSpider | ESISmartSpider | smart-spider.com |
| Earthcom | EARTHCOM.info/1.xbeta [www.earthcom.info] | Earthcom.info |
| Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu) | ||
| EARTHCOM.info/1.x [www.earthcom.info] | ||
| Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info) | ||
| EasyDL | EasyDL/3.04 http://keywen.com/Encyclopedia/Bot | Keywen |
| EasyDL/3.xx | ||
| EasyDL/3.xx http://keywen.com/Encyclopedia/Bot | ||
| Echo.com | Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 | Echo.com |
| Echo.fr | EchO!/2.0 | Echo.fr |
| Egothor | Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) | Charles University in Prague |
| Egotobot | EgotoBot/4.8 (+http://www.egoto.com/about.htm) | Egoto.com |
| Elfbot | elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) | uCHOOSE |
| Elsop | LinkScan/9.0g Unix | LinkScan |
| LinkScan/x.x Unix | ||
| LinkScan/11.0beta2 Unix | ||
| EmeraldShield.com Web Spider | EmeraldShield.com Web Spider (http://www.emeraldshield.com/webbot.aspx) | Emeraldshield |
| Enfish Tracker | Enfish Tracker | Enfish |
| Enoola | enoola (http://www.enoola.com) | Enoola |
| Enterprise Search | Enterprise_Search/1.0.xxx | Innerprise |
| Search/1.0 (http://www.innerprise.net/es-spider.asp) | ||
| Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp) | ||
| Enterprise_Search/1.0 | ||
| ES.NET_Crawler/2.0 (http://search.innerprise.net/) | ||
| Entireweb | WorldLight | Entireweb |
| Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com) | ||
| Speedy Spider (Beta/x.x; speedy@entireweb.com) | ||
| Speedy_Spider (http://www.entireweb.com) | ||
| Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/) | ||
| Envolkspider | envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) | Envolk |
| envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html) | ||
| envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) | ||
| EroCrawler | EroCrawler | EroCrawler |
| Eruvo-bot | eruvo-bot 4.8.1 (http://www.eruvo.com) | Eruvo |
| EuripBot | EuripBot/0.4 (+http://www.eurip.com) PreCheck | Eurip.com |
| EuripBot/0.2 (+http://www.eurip.com) GetRobots | ||
| EuripBot/0.4 (+http://www.eurip.com) GetFile | ||
| EuripBot/0.5 (+http://www.eurip.com) PreCheck | ||
| Euro-spider | Euro-Spider Shopping 1.0 | Euro-spider |
| Evaal | Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com) | Evaal |
| EvaalSE | EvaalSE – bot@evaal.com | Evaal |
| Eventax | eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) | Eventax |
| Everest-Vulcan | Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/cra wlerhelp) |
Vulcan |
| Exabot | Exabot/2.0 | Exalead |
| Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) | ||
| Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot) | ||
| Exalead NG/MimeLive Client (convert/http/0.120) | ||
| Exabot/3.0 | ||
| ExaBotTest/3.0 | ||
| ExaBotTest/2.0 | ||
| NG/2.0 | ||
| Exabot-Test/1.0 | ||
| Mozilla/5.0 (compatible; Konqueror/3.2; Linux) (KHTML, like Gecko) | ||
| Exabot-Images | Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot) | Exalead |
| Exabot-Images/1.0 | ||
| NG/4.0.1229 | ||
| ExactSEEK | ExactSeek Crawler/0.1 | ExactSEEK |
| exactseek.com | ||
| eseek-larbin_2.6.2 (crawler@exactseek.com) | ||
| exactseek-pagereaper-2.63 (crawler@exactseek.com) | ||
| exactseek-crawler-2.63 (crawler@exactseek.com) | ||
| ExactSeek_Spider | ExactSeek_Spider | ExactSeek |
| Excalibur | Excalibur Internet Spider V6.5.4 | Convera |
| Execrawl | Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) | Execrawl |
| FAST-WebCrawler | FAST-WebCrawler/2.1-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
FAST |
| FAST-WebCrawler/2.2-pre3 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.2-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support /crawler.asp) |
||
| FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre12 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsea rch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch /faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre4 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear ch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/ faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1.prealpha.2000-04-07.1 (ashen@looksmart.net) | ||
| FAST Enterprise Crawler/6.4.18 (crawler@fast.no) | ||
| FAST-SoccerCrawler/2.2-pre-cvs (oyvinda@fast.no; http://www.fast.no/faq/faqfastw ebsearch/faqfastwebcrawler.html) |
||
| FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net) | ||
| FAST-WebCrawler/2.1.pre.2000-04-18.1 (crawler@fast.no; http://www.fast.no/faq/fa qfastwebsearch/faqfastwebcrawler.html) |
||
| fastlwspider/1.0 | ||
| FAST-WebCrawler/2.1.pre.2000-04-14.1 (ashen@looksmart.net) | ||
| FDSE | Mozilla/4.0 (compatible; FDSE robot) | Abadoor |
| FaXobot | Faxobot/1.0 | Faxo |
| Factbot | factbot : http://www.factbites.com/robots | Factbites |
| Factbot 1.09 (see http://www.factbites.com/webmasters.php) | ||
| Fast Search | PycURL | FAST |
| Fastbot | fastbot crawler beta 2.0 (+http://www.fastbot.de) | Fastbot |
| Favo.eu crawler | favo.eu crawler/0.6 (http://www.favo.eu) | Favo |
| Feed24 | Feed24.com | Feed24 |
| FeedChecker | FeedChecker/0.01 | University of Tokyo |
| Feedfetcher-Google | Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) | |
| Feedster Crawler | Feedster Crawler/3.0; Feedster, Inc. | Feedster |
| Felix | Felix – Mixcat Crawler (+http://mixcat.com) | MixCat |
| Filangy | Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; fi langy-agent@filangy.com) |
Filangy |
| FindLinks | http://wortschatz.uni-leipzig.de/findlinks/ | University of Leipzig |
| findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/) | ||
| findlinks/1.1.1-a2 (+http://wortschatz.uni-leipzig.de/findlinks/) | ||
| findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/) | ||
| findlinks/0.901 (+http://wortschatz.uni-leipzig.de/findlinks/) | ||
| findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/) | ||
| findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) | ||
| Findexa Crawler | Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) | Findexa |
| FineBot | FineBot | Finesearch |
| Firefly | Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) | Fireball |
| Firefly/1.0 | ||
| FirstGov | FirstGov.gov Search – POC:firstgov.webmasters@gsa.gov | U.S.Government |
| Firstsbot | firstsbot | Firstsfind |
| Flapbot | Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor perio d com) |
Flaptor |
| Flatlandbot | great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www. flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com) |
Flatland Industries |
| great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlan dindustries.com/flatlandbot.php; jason@flatlandindustries.com) |
||
| flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindu stries.com/flatlandbot.php; jason@flatlandindustries.com) |
||
| FlickBot | FlickBot 2.0 RPT-HTTPClient/0.3-3 | DivX.com |
| Fluffy the spider | Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@se archhippo.com) |
Searchhippo |
| FnooleBot | FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) | Fnoole |
| Folkd.com Spider | Folkd.com Spider/0.1 beta 1 (www.folkd.com) | Folkd |
| ForAll.pl-Crawler | ForAll.pl-Crawler/1.0 | ForAll |
| Francis | Francis/1.0 (francis@neomo.de http://www.neomo.de/) | Neomo |
| FreshNotes crawler | FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com | FreshNotes |
| FreshNotes crawler, report problems to crawler-at-freshnotes-dot-com | ||
| Freshmeat | freshmeat.net URL validator/1.1 | Freshmeat |
| FuchsBot | FuchsBot +http://www.fuchsbot.tld | FuchsBot |
| FurlBot | Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn .furlbot@looksmart.net) |
Furl |
| FuseBulb | FuseBulb.Com | FuseBulb |
| FyberSpider | FyberSpider (+http://www.fybersearch.com/fyberspider.php) | FyberSearch |
| GAIS Robot | GAIS Robot/1.0B2 | Seed |
| GEXTEST-00393 | gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.co m) |
Unknown |
| GPU p2p crawler | Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engin e.php) |
GPU |
| GSiteCrawler | GSiteCrawler/v1.20 rev. 273 (http://gsitecrawler.com/) | GSiteCrawler |
| Gaaz | gazz/x.x (gazz@nttrd.com) | Infobee |
| Gaisbot | Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) | Gais |
| Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) | ||
| Gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) | ||
| GalaxyBot | Mozilla/4.0 (compatible; www.galaxy.com) | Galaxy |
| Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com/galaxybot.html) | ||
| GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html) | ||
| Gamekitbot | gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) | Uchoose |
| GammaSpider | GammaSpider/1.0 | Gammasite |
| GenieKnows | geniebot wgao@genieknows.com | GenieKnows |
| larbin_2.6.3 (wgao@genieknows.com) | ||
| Mozilla/5.0 wgao@genieknows.com | ||
| Mozilla/5.0 (wgao@genieknows.com) | ||
| GeonaBot | GeonaBot 1.x; http://www.geona.com/ | Geona |
| Georgia Institute of Technology | larbin_2.6.2 (listonATccDOTgatechDOTedu) | Georgia Institute of Technology |
| Geourl | Mozilla/5.0 (compatible; geourl/2.0b16 – http://geourl.org/bot) | Geourl |
| GigaBaz Brainbot | gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/) | Gigabaz |
| MicroBaz | ||
| Gigabot | Gigabot/2.0; http://www.gigablast.com/spider.html | Gigablast |
| Gigabot/3.0 (http://www.gigablast.com/spider.html) | ||
| Gigabot/2.0att | ||
| Gigabot/2.0/gigablast.com/spider.html | ||
| Gigabot/2.0 | ||
| Girafabot | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at giraf a dot com; http://www.girafa.com) |
Girafa |
| GlobalQueue | Look.com | Multi-mode |
| GnodSpider | GNODSPIDER (www.gnod.net) | Gnod |
| GoForIt | GOFORITBOT ( http://www.goforit.com/about/ ) | GoForIt |
| GoForIt.com | ||
| Goblin | Goblin/0.9 (http://www.goguides.org/) | GoGuides |
| Goblin/0.9.x (http://www.goguides.org/goblin-info.html) | ||
| Gonzo1 | gonzo1[P] +http://www.suchen.de/popups/faq.jsp | T-info |
| Gonzo2 | gonzo2[P] mailto:crawleradmin.t-info@telekom.de | T-info |
| gonzo1[P] mailto:crawleradmin.t-info@telekom.de | ||
| gonzo2[P] +http://www.suchen.de/faq.html | ||
| Goo (Japan) | Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Google-Adsense | Mediapartners-Google | |
| Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html) | ||
| Mediapartners-Google/2.1 | ||
| Google-Image | Googlebot-Image/1.0 | |
| Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html) | ||
| Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html) | ||
| Google-Sitemaps | Google-Sitemaps/1.0 | |
| Google-WAP | Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com | |
| Google WAP Proxy/1.0 | ||
| GoogleBot | Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) | |
| Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; http://www.google.com/bot.html) |
||
| Googlebot/Test ( http://www.googlebot.com/bot.html) | ||
| Googlebot/2.1 ( http://www.googlebot.com/bot.html) | ||
| Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Goog lebot/2.1; http://www.google.com/bot.html) |
||
| Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/) | ||
| Googlebot/1.0 (googlebot@googlebot.com) | ||
| Googlebot/2.1w (+http://googlebot.com/bot.html) | ||
| Googlebot/2.1 ( http://www.google.com/bot.html) | ||
| Googlebot/1.0 | ||
| Googlebot/2.0 (+http://googlebot.com/bot.html) | ||
| Googlebot-w/2.1 (+http://googlebot.com/bot.html) | ||
| Googlebot/2.0 beta (googlebot@googlebot.com) | ||
| Googlebot/2.1 (+http://www.googlebot.com/bot.html) | ||
| Googlebot/2.1 (+http://www.google.com/bot.html) | ||
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | ||
| Googlebot-Mobile | Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/b ot.html) |
|
| KDDI-CA33 UP.Browser/6.2.0.10.4 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
||
| Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Google bot-Mobile/2.1; +http://www.google.com/bot.html) |
||
| Greaterera | Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) | Greaterera |
| GrigorBot | GrigorBot 0.8 (http://www.grigor.biz/bot.html) | Grigor |
| Gromit | Gromit/1.0 | Australasian Legal Information Institute |
| Grub-client | Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://gru b.org) |
Grub |
| Gsa-crawler | gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) | IBM |
| gsa-crawler (Enterprise; GIX-04637; rex_li@trend.com.tw) | ||
| Gulliver | Gulliver/1.2 | Northernlight |
| Gulliver/1.3 | ||
| GulperBot | Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim /cgi-bin/Link/GulperBot) |
University of New-York |
| Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) | ||
| Gungho-crawler | Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) | Gungho |
| GurujiBot | GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) | Guruji |
| GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) | ||
| Harvest-NG | Harvest-NG/1.0.2 | Harvest-NG |
| Hatena Antenna | Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) | Hatena Antenna |
| HatenaScreenshot | HatenaScreenshot/1.0 (checker) | Hatena |
| HatenaScreenshot/1.0 (checker) | ||
| Hbtronix.spider | hbtronix.spider.2 — http://hbtronix.de/spider.php | Hbtronix |
| HeinrichderMiragoRobot | HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) | Mirago |
| Helix | Helix/1.x (+http://www.sitesearch.ca/helix/) | SiteSearch |
| HenriLeRobotMirago | HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) | Mirago |
| HenryTheMiragoRobot | HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) | Mirago |
| Heritrix | mozilla/5.0 (compatible; heritrix/1.3.0 +http://archive.crawler.org) | University of Washington |
| archive.org_bot | ||
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 +http://www.cs .washington.edu/research/networking/websys/) |
||
| Heritrix L3S | Mozilla/5.0 (compatible; heritrix/1.5.0 +http://www.l3s.de/~kohlschuetter/projec ts/crawling/) |
L3S Research Center |
| Heritrix/1.4.0 | Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) | Chepi |
| Hermits Search | Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) | Hermits Search |
| Hiiglespider | Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider | Hiigle |
| Hitwise Spider | Hitwise Spider v1.0 http://www.hitwise.com | Hitwise |
| Holmes | holmes/3.10.1 (OnetSzukaj/5.0; +http://szukaj.onet.pl) | Szukaj.onet |
| holmes/3.11 (http://morfeo.centrum.cz/bot) | ||
| holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl) | ||
| holmes/x.x | ||
| holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl) | ||
| HolmesBot | HolmesBot (http://holmes.ge) | Holmes |
| HomePageSearch | HomePageSearch(hpsearch.uni-trier.de) | HomePageSearch |
| Homerbot | Homerbot: www.homerweb.com | Homerweb |
| Honda-Search | Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda -search.com) |
Honda-Search |
| Hoowwwer | HooWWWer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at> hiit.fi) |
NGIR |
| HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto: crawler-info<at>hiit.fi) |
||
| Htdig | htdig/3.1.x (root@localhost) | ht:/dig |
| Htdig/3.1.6 | htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) | Acad?mie de Toulouse |
| I1searchbot | i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search .com) |
I1search |
| ICC-Crawler | ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-con tact(at)ml(dot)nict(dot)go(dot)jp) |
NICT |
| ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(a t)ml(dot)nict(dot)go(dot)jp) |
||
| ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-c ontact(at)ml(dot)nict(dot)go(dot)jp) |
||
| ICCrawler | ICCrawler – ICjobs (http://www.icjobs.de/bot.htm) | ICCenter |
| ICRA_Label_spider | ICRA_label_spider/x.0 | Icra |
| IDBot | Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) | Id-search |
| IIITBOT | IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr a t iiit dot ac dot in) |
Webkhoj |
| INGRID | Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://webmaster. ilse.nl/jsp/webmaster.jsp) |
Ilse |
| IP2MapBot | IP2MapBot/1.1 http://www.ip2map.com | Ip2Map |
| IPiumBot | IPiumBot laurion(dot)com | Laurions |
| IRLbot | IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) | Texas A&M University |
| IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) | ||
| IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler) | ||
| IWAgent | IWAgent/ 1.0 – www.brandprotect.com | Brandprotect |
| Iaskspider2 | iaskspider2 (iask@staff.sina.com.cn) | Sina |
| Ichiro | ichiro/1.0 (ichiro@nttr.co.jp) | Goo |
| ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) | ||
| ichiro/1.0 (ichiro@nttr.co.jp) | ||
| IconSurf | IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) | IconSurf |
| IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) | ||
| Icsbot | icsbot-0.1 | International Christian school of Seoul |
| Ideare | ideare – SignSite/1.x | Ideare |
| IlTrovatore | IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovator e.it) |
IlTrovatore |
| Ilial/Nutch | ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. Fo r more information please visit http://www.ilial.com/crawler; http://www.ilial.c om/crawler; crawl@ilial.com) |
Ilial |
| ilial/Nutch-0.9-dev | ||
| Ilse | Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmeld en.ilse.nl/?aanmeld_mode=webhints) |
Ilse |
| Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden. ilse.nl/?aanmeld_mode=webhints) |
||
| ImageWalker | ImageWalker/2.0 (www.bdbrandprotect.com) | Bdbrandprotect |
| IncyWincy | NetResearchServer/x.x(loopimprovements.com/robot.html) | LoopImprovements |
| IncyWincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovement s.com/robot.html) |
||
| IncyWincy(http://www.loopimprovements.com/robot.html) | ||
| IncyWincy/2.1(loopimprovements.com/robot.html) | ||
| IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovemen ts.com/robot.html) |
||
| IncyWincy (Look) | IncyWincy(http://www.look.com) | Look |
| IndexTheWeb | IndexTheWeb.com Crawler7 | IndexTheWeb |
| Indonesia Interactive | Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive |
Indonesia Interactive |
| InelaBot | InelaBot/0.2 (+http://inelegant.org/bot) | Inelegant |
| Inet Library | Inet library | Inet Library |
| InfoFly | InfoFly/1.0 (http://www.versions-project.org/) | Versions-project |
| InfoLab robot | Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) | Stanford University |
| InfoSec Search Bot | RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.ht ml; lhall@telegenetic.net) |
Telegenetic |
| Infoseek | InfoSeek Sidewinder/0.9 | Go |
| Inria | larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) | Inria |
| Insitor Search robot | Insitor.com search and find world wide! | Insitor |
| Insitornaut | Insitornaut | Insitor |
| Internet Ninja | Internet Ninja x.0 | Dream Train Internet |
| Internetseer | InternetSeer.com | Internetseer |
| Iprospect | Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) | Iprospect |
| IpselonBot | IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) | Ipselon |
| Iseekbot | iSEEKbot/iSEEKbot-0.9-dev (http://beta.iseek.com/iseekbot.html; bot at iseek dot com) |
Iseek |
| Ishida Lab | larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) | Kyoto University |
| It-bot | IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.ilt rovatore.it/bot.html; bot@iltrovatore.it) |
IlTrovatore |
| Jabot | Jabot/7.x.x (http://odin.ingrid.org/) | ODIN Directory |
| Jabot/6.x (http://odin.ingrid.org/) | ||
| Jambot | Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot ; crawler@jambot.com) |
Jambot |
| Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com) | ||
| Jayde Crawler | Jayde Crawler. http://www.jayde.com | Jayde |
| Jeanie | jeanie/3.3.3(www.sidedc.net/;compatible;MSIE 6.0;Windows NT 5.51) | Sidedc |
| Jetbot | Jetbot/1.0 | JetEye |
| Jobs.de-Robot | Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; jobsde@jobscout24.de) ( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesunge n/search/robot.html ) |
Neofonie |
| Jongaimpi | jongaimpi/2.10 (jonga; http://www.jonga.co.za; info@jonga.co.za) | Jonga |
| Jyxobot | Jyxobot/1 | Jyxo |
| Jyxobot/x | ||
| K2 Spider | k2spider | Verity |
| KAIST AITrc Crawler | KAIST AITrc Crawler | AITrc |
| KFSW-Bot | KFSW-Bot (Version: 1.01, powered by KFSW, www.kfsw.de) | KFSW |
| KIT_Fireball | KIT_Fireball/2.0 | Dino-online |
| KSbot | KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/conte nt/crawler/index.html; crawleradmin@knowledgestorm.com) |
Knowledgestorm |
| KakleBot | KakleBot – www.kakle.com/0.1 (KakleBot – www.kakle.com; http:// www.kakle.com/bo t.html; support@kakle.com) |
akle |
| KaloogaBot | kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; craw ler@kalooga.com) |
kalooga |
| kalooga/KaloogaBot (Kalooga; http://www.kalooga.com; info@kalooga.com) | ||
| Kasparek | Firefox_1.0.6 (kasparek@naparek.cz) | Czech Technical University Prague |
| Keegeebot | Keegeebot/2.1 (+http://www.keegee.com/keegee/bot.html) | Keegee |
| Kenjin Spider | Kenjin Spider | Kenjin |
| Kevin | Kevin http://dznet.com/kevin/ | Dznet.com |
| Kevin http://websitealert.net/kevin/ | ||
| KicktooBot | kicktooBotV1.1 kictooBot@kictoo.com | Kicktoo |
| Kinja-imagebot | kinja-imagebot (http://www.kinja.com/) | Kinja |
| Kinjabot | kinjabot (http://www.kinja.com) | Kinja |
| KnowItAll | KnowItAll(knowitall@cs.washington.edu) | University of Washington |
| Knowledge.com | Knowledge.com/0.x | knowledge.com |
| Krugle | Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info .html; webcrawler@krugle.com) |
Krugle |
| Kulokobot | kulokobot www.kuloko.com kuloko@backweave.com | Kuloko |
| kuloko-bot/0.x | ||
| Kulturarw | kulturarw3/0.1 | National Library of Sweden |
| Kumm | KummHttp/1.1 (compatible; KummClient; Linux rulez) | Sanoma |
| Kyluka crawl | Mozilla/5.0 (compatible; Kyluka crawl; http://www.kyluka.com/crawl.html; crawl@k yluka.com) |
Kyluka |
| LECodeChecker | LECodeChecker/3.0 libgetdoc/1.0 | Linkexchange |
| LNSpiderguy | LNSpiderguy | Lexis-Nexis |
| LapozzBot | LapozzBot/1.5 (+http://robot.lapozz.hu) | Lapozz |
| LapozzBot/1.4 (+http://robot.lapozz.hu) | ||
| LapozzBot/1.4 ( http://robot.lapozz.com) | ||
| Larbin_2.6.3 | larbin_2.6.3 larbin2.6.3@unspecified.mail | Unknown |
| larbin_2.6.3 marzia.polito@intel.com | ||
| Lawinfo-crawler | lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo .com; webmaster@lawinfo.com) |
Lawinfo |
| Lemur Consulting | larbin_2.6.2 (tom@lemurconsulting.com) | Lemur Consulting |
| Lexibot | Mata Hari/2.00 | BrightPlanet |
| LexiBot/1.00 | ||
| Liafa | larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) | Liafa |
| LibWeb | libWeb/clsHTTP — hiongun@kt.co.kr | Korea Telecom |
| LibertyW | LibertyW (+http://www.lw01.com) | LibertyW |
| LibertyW (+http://www.libertyw.eu) | ||
| LijitSpider | LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)co m) |
Lijit |
| LinkWalker | LinkWalker | Seven TwentyFour |
| Linknzbot | linknzbot | LinkNZ |
| Links2Go | Mozilla/3.01 (Compatible; Links2Go Similarity Engine) | Links2Go |
| Links4US-Crawler | Links4US-Crawler, (+http://links4us.com/) | Links4US |
| LinksManager.com_bot | Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkcheck er.html) |
Unknown |
| Llaut | Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) | Universitat de les Illes Balears |
| Lmspider | lmspider (lmspider@scansoft.com) | Nuance |
| LocalBot | LocalBot/1.0 ( http://www.localbot.co.uk/) | LocalBot |
| Lockstep Spider | Lockstep Spider/1.0 | Lockstep |
| Look.com | NetResearchServer(http://www.look.com) | |
| LookdirBot | LookdirBot | Lookdir |
| Lovel | Lovel as 1.0 ( +http://www.everatom.com) | Everatom |
| Ltaa_web_crawler | larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) | Ecole Polytechnique F?d?rale de Lausanne |
| Luchs.at URL checker | luchs.at URL checker | Luchs |
| Lycos_Spider | Lycos_Spider_(modspider) | Lycos |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_(T-Rex) ; Lycos_Spide r_(T-Rex) ) |
||
| Mozilla/4.0 (compatible; MSIE 5.0; Windows 98;Lycos_Spider_Beta2(T-Rex) ; Lycos_ Spider_Beta2(T-Rex) ) |
||
| Lycos_Spider_(T-Rex) | ||
| MJ12bot | MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) | Majestic 12 |
| MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+) | ||
| MJ12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+) | ||
| Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+) | ||
| MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) | ||
| MJ12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+) | ||
| MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+) | ||
| MQBOT | MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.ui uc.edu) |
University of Illinois |
| MSN Bot | msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) | MSN |
| msnbot/1.1 (+http://search.msn.com/msnbot.htm) | ||
| msnbot/0.3 (+http://search.msn.com/msnbot.htm) | ||
| msnbot/0.9 (+http://search.msn.com/msnbot.htm) | ||
| msnbot/1.0 (+http://search.msn.com/msnbot.htm) | ||
| msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm) | ||
| msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | ||
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) | ||
| MSNBOT_Mobile | MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) | MSN |
| MSRBOT | MSRBOT (http://research.microsoft.com/research/sv/msrbot) | Microsoft |
| MSRBot | MSRBOT (http://research.microsoft.com/research/sv/msrbot/) | Microsoft |
| MSRBOT (http://research.microsoft.com/research/sv/msrbot/ | ||
| MaSagool | MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) | Sagool |
| Mail.Ru | Mail.Ru/1.0 | Mail.Ru |
| Mainseek_Bot | Mozilla/5.0 (compatible;MAINSEEK_BOT) | Mainseek |
| Mammoth | Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1 | SLI Systems |
| mammoth/1.0 (+http://www.sli-systems.com/) | ||
| Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1 | ||
| MantraAgent | MantraAgent | LookSmart |
| Mariner | Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) | Kolibri |
| Martini | Martini | LookSmart |
| MARTINI | ||
| Marvin | Marvin v0.3 | Health On the Net Fondation |
| Masterseek | MasterSeek | Masterseek |
| Maxbot | Spider/maxbot.com admin@maxbot.com | Maxbot |
| Maxomobot | maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com) |
Maxomo |
| MediaCrawler | MediaCrawler-1.0 (Experimental) | Media Find |
| MediaSearch | MediaSearch/0.1 | WWW.FI |
| Mediater Rechercher | libwww/5.3.2 | Mediater |
| MegaSheep | MegaSheep v1.0 (www.searchuk.com internet sheep) | Search UK |
| Megaglobe Crawler | Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com) | Megaglobe |
| Melbot WebSpider | Melbot WebSpider & RSS News Crawler www.melbot.info (V.2.42 by A.I.C.E.) | Melbot |
| Mercator | Mercator-1.x | Altavista |
| Mercator-Scrub-1.1 | ||
| Mercator-2.0 | ||
| Merl.com | larbin_2.1.1 larbin2.1.1@somewhere.com | Mitsubishi Electrical Research Lab |
| MetaGer_PreChecker | MetaGer_PreChecker0.1 | MetaGer |
| Metacarta | Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com) | Metacarta |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com) | ||
| Metaeuro Web Crawler | Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.meta euro.com; crawler at metaeuro dot com) |
Metaeuro |
| Metager-Linkchecker | MetaGer-LinkChecker | Metager |
| MetagerBot | MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) | Metager |
| Metaquerier | MQbot http://metaquerier.cs.uiuc.edu/crawler | University of Illinois |
| MQbot metaquerier.cs.uiuc.edu/crawler | ||
| Metaspinner | Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner .de/) |
Metaspinner |
| Metatagsdir | metatagsdir/0.7 (+http://metatagsdir.com/directory/) | Metatagsdir |
| Microsoft Small Business Indexer | Microsoft Small Business Indexer | Microsoft |
| Microsoft URL Control | Microsoft URL Control – 6.00.8862 | Unknown |
| Microsoft URL Control – 6.01.9782 | ||
| Miggibot | A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24 |
Micro-sys |
| Mirar | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Mirar Search Indexing Agent s ee getmirar.com) |
Mirar |
| Missigua Locator 1.9 | Missigua Locator 1.9 | Unknown |
| Misterbot | Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot .fr) |
Misterbot |
| Miva | Miva (AlgoFeedback@miva.com) | Miva |
| MnoGoSearch | MnogoSearch/3.2.xx | MnoGoSearch |
| Mo College 1.9 | Mo College 1.9 | Unknown |
| Moget | moget/x.x (moget@goo.ne.jp) | Goo |
| Mogimogi | mogimogi/1.0 | Goo |
| MojeekBot | Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) | Mojeek |
| MojeekBot/0.x (archi; http://www.mojeek.com/bot.html) | ||
| Monogol | http://www.monogol.de | Monogol |
| Moreoverbot | Moreoverbot/5.00 (+http://www.moreover.com) | Moreover |
| Morris | Morris – Mixcat Crawler (+http://mixcat.com) | MixCat |
| Mowserbot | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowserbot; http://www.mowser. com/bot) |
Mowser |
| MozDex | mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) | MozDex |
| Mozdex | Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com) | MozDex |
| MultiText | MultiText/0.1 | Virginia Polytechnic Institute and State University |
| Multicrawler | multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) | SWSE |
| multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html) | ||
| MuscatFerret | Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com) | Euro Ferret |
| Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com) | ||
| Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk) | ||
| MusicWalker | MusicWalker2.0 (+http://www.somusical.com) | SoMusical! |
| My-bytebot | my-bytebot/1.0 (+http://spider.my-byte.de/info) | My-byte |
| MyFamilyBot | Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.ancestry.com/learn/bot.aspx ; SearchBot@MyFamilyInc.com) |
Ancestry |
| Mylinea | Mylinea.com Crawler 2.0 | Mylinea |
| NASA Search 1.0 | NASA Search 1.0 | Unknown |
| NCSA | NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) | National Center for Supercomputing Applications |
| NG-SearchBot | NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) | NG-Search |
| NG-Search/0.9.8 (http://www.ng-search.com) | ||
| NII | larbin_2.6.2 (hamasaki@grad.nii.ac.jp) | National Institut of Informatics (Japan) |
| NLCrawler | Mozilla/5.0 (compatible; NLCrawler/2.0.27; Linux 2.6.3-7; i686; en_US)KHTML/3.4. 89 (like Gecko) |
Northern Light |
| NMG Spider | NMG Spider/0.3 (szukanko.com) | Szukanko |
| NTT Directory | nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) | NTT Directory |
| NWSpider | NWSpider 0.9 | NWSpider |
| Nabot | nabot_1.0 | Naver |
| NABOT/5.0 | ||
| Najdi | Mozilla/5.0 (compatible; InterseekWeb/3.x) | Najdi |
| Nameprotect | NPBot (http://www.nameprotect.com/botinfo.html) | Nameprotect |
| NPBot-1/2.0 | ||
| NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com) | ||
| Nationaldirectory | NationalDirectoryAddURL/1.0 | Nationaldirectory |
| NationalDirectory-WebSpider/1.3 | ||
| NatzanBot | NatzanBot, http://www.natzan.com | Natzan |
| NaverBot | NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) | Naver |
| NaverBot_dloader/1.5 | ||
| dloader(NaverRobot)/1.0 | ||
| NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com) | ||
| Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp) | ||
| NavissoBot | NavissoBot/1.7 (+http://navisso.com/) | Navisso |
| NavissoBot | ||
| Nebulla | Nebullabot/2.2 (http://bot.nebulla.info) | Nebulla |
| Nebullabot/2.2 (http://bot.nebulla.de) | ||
| Nebulla/V1.0 Spider (http://spider.nebulla.de) | ||
| Nelian Pty Ltd – Spider | Nelian Pty Ltd – Spider v2.1 ( http://pcaccessoriesparts.com ) | Pcaccessoriesparts |
| NetNose | Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com) |
NetNose |
| NetResearchServer | NetResearchServer/4.0(loopimprovements.com/robot.html) | Loopimprovements |
| NetSprint — 2.0 | NetSprint — 2.0 | Wirtualna Polska |
| NetWhatCrawler | NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com ; support@netwhat.com) |
Netwhat |
| NetinfoBot | NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) | NetinfoBot |
| Netluchs | Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don’t___spam_me_@netluchs.de) | Netluchs |
| Netprospector | Netprospector JavaCrawler | Netprospector |
| Netscape | Robozilla/1.0 | DMOZ |
| NextGenSearchBot | NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGe nSearchBot.aspx) |
Zoominfo |
| NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot ) |
||
| NextGenSearchBot 1 (for information visit http://www.zoominfo.com/NextGenSearchB ot) |
||
| NextopiaBot | NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v1.1 | Nextopia |
| NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x | ||
| NimbleCrawler | Mozilla/5.0 (Windows;) NimbleCrawler 2.0.0 obeys UserAgent NimbleCrawler For pro blems contact: crawler@healthline.com |
Healthline |
| Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For prob lems contact: crawler@healthline.com |
||
| Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obe ys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com |
||
| Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For pro blems contact: crawler@healthline.com |
||
| Noago Spider | Noago Spider | Noago |
| NokodoBot | NokodoBot/1.x (+http://nokodo.com/bot.htm) | Nokodo |
| Norbert | Norbert the Spider(Burf.com) | Burf |
| Noxtrumbot | noxtrumbot/1.0 (crawler@noxtrum.com) | Noxtrum |
| Noyona | noyona_0_1 | Noyona |
| Nsyght | nsyght.com/Nutch-0.9 (nsyght.com; search.nsyght.com) | Nsyght |
| NuSearch Spider | nuSearch Spider <a href=http://www.nusearch.com>www.nusearch.com</a> (compatible ; MSIE 4.01) |
NuSearch |
| Nutch | Nutch | Lucene |
| NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists. sourceforge.net) |
||
| NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists. sourceforge.net) |
||
| NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lis ts.sourceforge.net) |
||
| NutchCVS/0.8-dev | NutchCVS/0.8-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sy crawl@cs.washington.edu) |
University of Washington |
| NutchEC2Test | NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.o rg/nutch/bot.html; ec2test at lucene.com) |
Amazon |
| Obidos-bot | obidos-bot (just looking for books.) | Weblog bookwatch |
| Object Sciences Corp. | Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3 E |
|
| Objects Search | ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; sup port@thesoftwareobjects.com) |
SAIC |
| Ocelli | Ocelli/1.x (http://www.globalspec.com/Ocelli) | GlobalSpec |
| Ocelli/1.3 (http://www.globalspec.com/Ocelli) | ||
| Octora | Octora Beta – www.octora.com | Octora |
| Octora Beta Bot | Octora Beta Bot – www.octora.com | Octora |
| OmniExplorer | OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet Categorizer | OmniExplorer |
| OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler | ||
| OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer | ||
| OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler | ||
| OnetSzukaj | Onet.pl SA, http://szukaj.onet.pl | Szukaj |
| Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl) | ||
| OpenISearch | OpenISearch/1.x (www.openisearch.com) | OpenISearch |
| OpenPortal4U | DigOut4U | Arisem |
| OpenTaggerBot | OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) | OpenTagger |
| OpenTextSiteCrawler | OpenTextSiteCrawler/2.9.2 | OpenText |
| OpenWebSpider | OpenWebSpider/x | OpenWebSpider |
| Openfind | Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.ht ml) |
Openfind |
| Openfind Robot/1.1A2 | ||
| Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www. openfind.com.tw/robot.html) |
||
| OpidooBot | OpidooBOT (larbin2.6.3@unspecified.mail) | Opidoo |
| Oracle | Oracle Ultra Search | Oracle |
| OrangeSpider | OrangeSpider | Orangeslicer |
| Orbiter | Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) | DailyOrbit |
| Overture-WebCrawler | Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/suppor t/crawler.asp) |
FAST |
| OzMonitor | Mozilla/5.0 (Linux; fr) Firefox (ozMonitor Free) | OzMonitor |
| Ozelot | ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat .de) |
Flying-cat |
| PJspider | PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) | Nextopia |
| PWeBot | PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) | Programacionweb |
| Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php) | ||
| Page-store | Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) | Page-store |
| PageBitesHyperBot | PageBitesHyperBot/600 (http://www.pagebites.com/) | PageBites |
| Page_verifier | page_verifier http://www.securecomputing.com/goto/pv | Securecomputing |
| Pagebull | Pagebull http://www.pagebull.com/ | Pagebull |
| Pages Jaunes | FAST Enterprise Crawler 6 used by Pages Jaunes (crawladmin@gmail.com) | Pages Jaunes |
| FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) | ||
| FAST Enterprise Crawler 6 used by Pages Jaunes (fastadmin@pagesjaunes.fr) | ||
| PalmeraBot | Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Versi on 0.001 |
Links24h |
| Pandora | Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 +http://pandora.nla.gov.au/ crawl.html) |
National Library of Australia |
| ParaSite | ParaSite/1.0b (http://www.ianett.com/parasite/) | Ianett |
| Patwebbot | Patwebbot (http://www.herz-power.de/technik.html) | Patsearch |
| Peerbot | PEERbot www.peerbot.com | Peerbot |
| Petitsage | www.petitsage.fr site detector 0.4 | Petitsage |
| PicoSearch | PicoSearch/1.0 | Pico Search |
| PictureOfInternet | PictureOfInternet/3.0 (http://www.malfunction.org/poi; erik@malfunction.org) | Malfunction |
| Piffany | Piffany_Web_Spider_v0.x | Piffany |
| Piffany_Web_Scraper_v0.x | ||
| Pilot Hitlist | Marketwave Hit List | Pilot Hitlist |
| Pingdom | Pingdom GIGRIB v1.1 (http://www.pingdom.com) | Pingdom |
| PipeLiner | pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) | Pipeline |
| Pita | Pita | Stanford University |
| PlantyNet_WebRobot | PlantyNet_WebRobot_V1.9 dhkang@plantynet.com | PlantyNet |
| PlantyNet_WebRobot_V1.9 yangsam@plantynet.com | ||
| PluckFeedCrawler | PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers) |
Pluck |
| Pmoz | Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botin fo.htm) |
Pmoz |
| Podtech | Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) | Podtech |
| Pogodak | Mozilla/5.0 (compatible; pogodak.ba/3.x) | Pogodak |
| Polybot | polybot 1.0 (http://cis.poly.edu/polybot/) | Polytechnic University Brooklyn |
| Pompos | Pompos | Iliad (Free) |
| Pompos/1.3 http://dir.com/pompos.html | ||
| Pompos/1.x pompos@iliad.fr | ||
| Pompos/1.1 http://dir.com/pompos.html | ||
| Pompos/1.2 http://dir.com/pompos.html | ||
| Pompos/1.x http://dir.com/pompos.html | ||
| PopJapanSearch | Robot/www.pj-search.com | PopJapanSearch |
| Popdexter | Popdexter/1.0 | Popdex |
| PrivacyFinder/1.1 | PrivacyFinder/1.1 | AT&T Privacy Bird Privacy Preferences |
| ProWebguide | ProWebGuide Link Checker (http://www.prowebguide.com) | ProWebguide |
| Probe! | PROBE! (Probert Encyclopaedia Research Robot V1.0 http://www.probertencyclopaedi a.com) |
Probert Encyclopaedia |
| Psbot | psbot/0.1 (+http://www.picsearch.com/bot.html) | Picsearch |
| Psycheclone | psycheclone | Unknown |
| Pubblisito | info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca | Pubblisito |
| Pumpkin | blogsearchbot-pumpkin-3 | Artofcomputing |
| Python-urllib | Python-urllib/1.16 | Unknown |
| QPCreep | QPCreep Test Rig ( We are not indexing, just testing ) | Quepasa |
| QihooBot | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.n et) |
Qihoo |
| Quantcastbot | Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) | Quantcast |
| Qube | qube/2.0 (+http://qube.qelix.com/v2) | Qube |
| QuepasaCreep | QuepasaCreep v0.9.1x | Quepasa |
| QuepasaCreep ( crawler@quepasacorp.com ) | ||
| QueryN Metasearch | QueryN Metasearch | QueryN |
| Qweerybot | QweeryBot/3.01 ( http://qweerybot.qweery.nl) | Qweery |
| Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com) | ||
| QweeryBot/3.02 ( http://qweerybot.qweery.nl) | ||
| R6_CommentReader | R6_CommentReader_(www.radian6.com/crawler) | Radian6 |
| R6_FeedFetcher | R6_FeedFetcher_(www.radian6.com/crawler) | Radian6 |
| RAMPyBot | RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lu cene.apache.org) |
GiveRamp |
| RAMPyBot – www.giveRAMP.com/0.1 (RAMPyBot – www.giveRAMP.com; http://www.giveram p.com/bot.html; support@giveRAMP.com) |
||
| RSS One Engine | RSS One Engine/0.72 (+http://www.rss-one.com) | RSS One |
| Rambot | ramBot xtreme x.x | Intersearch |
| RankMeter | Web Snooper | SearchUtilities |
| Rational SiteCheck | Rational SiteCheck (Windows NT) | GSInnova |
| Reaper | Reaper/2.0x (+http://www.sitesearch.ca/reaper) | Marty Anstey |
| Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/) | ||
| RedBot | RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) | Rediff |
| RedCarpet/1.2 | RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) | Pronto |
| RedCell | RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) | Telegenetic |
| RedKernel | RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) | RedKernel |
| RoboPal | RoboPal (http://www.findpal.com/) | FindPal |
| Rotondo | Rotondo/3.1 libwww/5.3.1 | QualiGo |
| RufusBot | RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) | Webaroo |
| RutterBot | RutterBot(+http://www.aktienbetreuer.de/bot.html) | Aktienbetreuer |
| SAIT | sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) | Samsung |
| SBIder | SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.site sell.com/contact-support.html) |
Sitesell |
| SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html) | ||
| SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell .com/contact-support.html) |
||
| SandCrawler | SandCrawler – Compatibility Testing | Microsoft |
| Savvybot | savvybot/0.2 | WebSavvy |
| ScanWeb | ScanWeb | Eserver |
| Schibstedsokbot | schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http:/ /www.schibstedsok.no/bot/) |
Schibsted |
| ScholarUniverse | ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scho laruniverse.com) |
ScholarUniverse |
| Scirus-crawler | FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com /srsapp/contactus/ |
Scirus |
| ScollSpider | ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) | Webwobot |
| Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com) | ||
| ScorpionBot | Fooky.com/ScorpionBot/ScoutOut;http://www.fooky.com/scorpionbots | Fooky |
| ScoutAnt | ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ | Ant |
| Scrubby | Scrubby/2.x (http://www.scrubtheweb.com/) | Scrub the web |
| Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/) | ||
| Sdcresearchlabs-testbot | sdcresearchlabs-testbot/0.8-dev (www.shopping.com/bot.html; http://lucene.apache .org/nutch/bot.html; researchbot@shopping.com) |
Shopping |
| Search-Engine-Studio | Search-Engine-Studio | Xtreeme |
| Search-Info | Spider-Sleek/2.0 (+http://search-info.com/linktous.html) | Search-Info |
| Search.ch | search.ch V1.4 | Search CH |
| search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) | ||
| Search4free | lwp-trivial/1.34 | Search4free |
| SearchByUsa | SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUs a.com) |
Search4USA |
| SearchEngineWorlds | ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ) | SearchEngineWorlds |
| SearchExpress Spider | SearchExpress Spider0.99 | SearchExpress |
| SearchScout | ClariaBot/1.0 | SearchScout |
| Diamond/1.0 | ||
| DiamondBot | ||
| SearchSight | SearchSight/2.0 (http://SearchSight.com/) | SearchSight |
| SearchSpider | Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspi der.com) |
SearchSpider |
| SearchSpider.com/1.1 | ||
| SearchdayBot | SearchdayBot | Searchday |
| Searchguild | SearchGuild_DMOZ_Experiment (chris@searchguild.com) | SearchGuild |
| Searchit-Now Robot | Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) | Searchit-Now |
| Searchmee! Spider | Searchmee! Spider v0.98a | Searchmee! |
| Seekbot | Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 | SeekPort |
| Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2 | ||
| Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3 | ||
| Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF) | ||
| Seeker.lookseek | Seeker.lookseek.com | Lookseek |
| Semager | Semager/1.0 (http://www.semager.de) | Semager |
| Semager/1.1 (http://www.semager.de/blog/semager-bots/) | ||
| Sensis.com.au Web Crawler | Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau) | Sensis |
| SeznamBot | SeznamBot/2.0-test (+http://fulltext.sblog.cz/) | Seznam |
| SeznamBot/1.0 (+http://fulltext.seznam.cz/) | ||
| SeznamBot/1.0 | ||
| SharewarePlaza | Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) | SharewarePlaza |
| Sherlock | sherlock/1.0 | Indiana University School of Informatics |
| Shim-Crawler | Shim-Crawler(NICT)(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl er/; crawler@logos.ic.i.u-tokyo.ac.jp) |
Chikayama-Taura laboratory |
| Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; c rawl@logos.ic.i.u-tokyo.ac.jp) |
||
| Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawl/; cra wl@logos.ic.i.u-tokyo.ac.jp) |
||
| ShopWiki | ShopWiki/1.0 ( +http://www.shopwiki.com/) | Littlewiki |
| ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) | ||
| Shoula | Shoula.com Crawler 2.0 | Shoula! Search |
| SietsCrawler | SietsCrawler/1.1 (+http://www.siets.biz) | Siets |
| Siigle Orumcex | Siigle Orumcex v.001 Turkey (http://www.siigle.com) | Siigle |
| Silk/1.0 | silk/1.0 | Slider |
| silk/1.0 (+http://www.slider.com/silk.htm)/3.7 | ||
| Sirketcebot | Sirketcebot/v.01 (http://www.sirketce.com/bot.html) | Sirketce |
| Site Server 3.0 Robot | Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR | American College of Radiology |
| SiteBar | SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/) | SiteBar |
| SiteBaseBot | Mozilla/5.0 (compatible; SiteBaseBot/1.0; http://dir.sitebase.ru) | SiteBase |
| SiteSpider | SiteSpider +(http://www.SiteSpider.com/) | SiteSpider |
| SiteTruth | SiteTruth.com site rating system | SiteTruth |
| SitiDiBot | +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) | SitiDi.net |
| Skampy | Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] | Skaffe |
| Skizzle | User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 – www.SKIZZLE .com) |
Skizzle |
| SkreemRBot | Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) | SkreemR |
| Slider | Slider_Search_v1-de | Slider |
| Slurp (Yahoo) | Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/ slurp) |
Yahoo |
| Slurp China (Yahoo) | Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) | Yahoo |
| Slurp Inktomi | Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | Hotbot-Lycos,NBCi etc. |
| Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | ||
| Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | ||
| Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | ||
| Slurp Inktomi (Yahoo) | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slu rp) |
Yahoo |
| Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.5) Gecko/2007071 3 Firefox/2.0.0.5 |
||
| Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch /slurp) |
||
| Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20071214 BonE cho/2.0.0.4 |
||
| Slurpy Verifier | Slurpy Verifier/1.0 | Yahoo |
| SmiffyDCMetaSpider | SmiffyDCMetaSpider/1.0 | Smiffysplace |
| Snapbot | Snapbot/1.0 | Hotlinks |
| Snoopy | Snoopy v1.2 | Unknown |
| SnykeBot | SnykeBot/0.6 (http://www.snyke.com) | Snyke |
| SoftHypermarket | SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) | SoftHypermarket |
| Sogou web spider | Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) | Sogou |
| Sohu-search | sohu-search | Sohu |
| Somewhere | Mozilla (Mozilla@somewhere.com) | Somewhere |
| Spam Bot | Mozilla/2.0 (compatible; NEWT ActiveX; Win32) | Unknown |
| Speedfind | speedfind ramBot xtreme 8.1 | Lotse |
| Speedy Spider | Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/s peedyspider/) |
Entireweb |
| Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/s peedyspider/) |
||
| Speedy Spider (Beta/1.0; www.entireweb.com) | ||
| Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/s peedyspider/) |
||
| Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/) | ||
| Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/s peedyspider/) |
||
| Sphere Scout | Sphere Scout&v4.0 – scout at sphere dot com | Sphere |
| Sphsearch | FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg ) |
Singapore Press Holdings |
| SpiderMonkey | Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) | SpiderMonkey |
| SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml) | ||
| SplatSearch | libwww-perl/5.45 | Splat |
| Spock Crawler | Spock Crawler (http://www.spock.com/crawler) | Spock |
| Sproose | sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@spr oose.com) |
Sproose |
| sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.c om) |
||
| StackRambler | StackRambler/2.0 | Rambler |
| StackRambler/x.x | ||
| StackRambler/2.0 (MSIE incompatible) | ||
| Steeler | Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) | University of Tokyo |
| Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) | ||
| Strategic Board Bot | Strategic Board Bot (+http://www.strategicboard.com) | Strategic Board |
| Suchbaer | suchbaer.de (CrawlerAgent v0.103) | Suchbaer |
| suchbaer.de | ||
| Suchpad | suchpad/1.0 (+http://www.suchpad.de) | Suchpad |
| SummizeBot | Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) | Summize |
| SuperSnooper | Robot@SuperSnooper.Com | SuperSnooper |
| SurveyBot | SurveyBot/2.3 (Whois Source) | Domaintools |
| Susie | !Susie (http://www.sync2it.com/susie) | Sync2it |
| Swooglebot | Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.html) | University of Maryland |
| SycaBoT | SycaBoT/1.0 | SycaroX |
| SycaBoT-Audio | SycaBoT-Audio | SycaroX |
| SycaBoT-Image | SycaBoT-Image | SycaroX |
| SycaBoT-Programme | SycaBoT-Programme | SycaroX |
| SycaBoT-Video | SycaBoT-Video | SycaroX |
| Sycrawl | NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycr awl@cs.washington.edu) |
University of Washington |
| Sygol | <http://www.sygol.com/> http://www.sygol.com | Sygol |
| SygolBot | SygolBot http://www.sygol.com | Sygol |
| SygolBot http://www.sygol.net | ||
| SynooBot | SynooBot/Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot. html) |
Synoo |
| Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html) | ||
| Syntryx | Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler | Syntryx |
| Szukacz | Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl) | Szukacz |
| Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl) | ||
| Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl) | ||
| TAGword | Tagword (http://tagword.com/dmoz_survey.php) | TAGword DMOZ survey |
| TCDBOT | TCDBOT/Nutch-0.8 (PhD student research;”http://www.tcd.ie; mcgettrs at t c d dot IE)” |
Trinity College Dublin |
| Tags2dir | tags2dir.com/0.8 (+http://tags2dir.com/directory/) | Tags2dir |
| TailRank Robot | Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:TailRank; http://ta ilrank.com/robot) Gecko/20021130 |
TailRank |
| Talkro | Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com, Home: http://222.122.15.190/w ebshot) |
DaumSoft |
| TargetSeek | Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/ TargetSeek.html) |
Targetgroups |
| Technoratibot | Technoratibot/0.7 | Technorati |
| Tecomi Bot | Tecomi Bot (http://www.tecomi.com/bot.htm) | Tecomi |
| Teemer | Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://w ww.netseer.com/crawler.html; crawler@netseer.com) |
Netseer |
| TelenetDigger | TelenetDigger/1.0; htdig/3.1.6 (webdev@staff.telenet.be) | Telenet |
| Teoma | teoma_agent1 | Teoma |
| Teoma MP | ||
| teomaagent crawler-admin@teoma.com | ||
| teomaagent1 [crawler-admin@teoma.com] | ||
| Mozilla/2.0 (compatible; Ask Jeeves/Teoma) | ||
| Teradex Mapper | Teradex Mapper; mapper@teradex.com; http://www.teradex.com | Teradex |
| TeragramCrawler | TeragramCrawler | Teragram |
| TeragramWebcrawler | TeragramWebcrawler/1.0 | Teragram |
| TerrawizBot | TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) | Terrawiz |
| TheSuBot | TheSuBot/0.1 (www.thesubot.de) | TheSuBot |
| Theme Spider | Theme Spider (+http://www.themespider.com/spider.html) | Themespider |
| Theophrastus | Mozilla/5.0 (compatible; Theophrastus/1.1; http://users.cs.cf.ac.uk/N.A.Smith/th eophrastus.php) |
N.A.Smith |
| Thumbshots-de-bot | thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) | Thumbshots |
| Thunderstones Webinator | Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) | Thunderstones |
| TimboBot | timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html | Breakingblogs |
| TivraSpider | tivraSpider/1.0 (crawler@tivra.com) | AT&T |
| Tkensaku | Tkensaku/x.x(http://www.tkensaku.com/q.html) | Tkensaku |
| ToileBot | Mozilla/5.0 (+http://www.toile.com/) ToileBot/0.1 | La Toile du Qu?bec |
| Topodia | Topodia/1.2-dev (Topodia – Crawler for HTTP content indexing; http://www.topodia .com/; support@topodia.com) |
Topodia |
| Toutatis | Toutatis x-xx.x (hoppa.com) | Hoppa |
| Toutatis x.x (hoppa.com) | ||
| Toutatis x.x-x | ||
| Traazibot | traazibot/testengine (+http://www.traazi.de) | Traazi |
| Trampelpfad | Trampelpfad-Spider | Trampelpfad |
| Trampelpfad-Spider-v0.1 | ||
| TrendTech | http://www.trendtech.dk/spider.asp) | TrendTech |
| Truveo | Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) | Truveo |
| Turnitin | TurnitinBot | Turnitin |
| TurnitinBot | TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) | Turnitin |
| Turnpike Emporium | Turnpike Emporium LinkChecker/0.1 | Turnpike Emporium Directory |
| TutorGigBot | TutorGig/1.5 (+http://www.tutorgig.com/crawler) | TutorGig |
| Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) | ||
| Twiceler | Twiceler www.cuill.com/twiceler/robot.html | Cuill |
| Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html) | ||
| Twiceler-0.9 http://www.cuil.com/twiceler/robot.html | ||
| Twiceler www.cuil.com/twiceler/robot.html | ||
| Twiceler-0.9 http://www.cuill.com/twiceler/robot.html | ||
| Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) | ||
| TygoBot | TygoProwler | Tygo |
| TygoBot | ||
| TÜzilla | Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuez illa.de/t_st-odp-entries-agent.html) |
TÜzilla |
| UKWizz | Mackster( http://www.ukwizz.com ) | UKWizz |
| UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/) | ||
| UN OCHA | FAST Enterprise Crawler 6 used by suckling@un.org (UN OCHA) | UN OCHA |
| UOLCrawler | UOLCrawler (soscrawler@uol.com.br) | UOL |
| URL-Spider | Mozilla/5.0 URL-Spider | URL-Spider |
| URL_Spider_Pro | URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) | Innerprise |
| URL_Spider_Pro/x.x | ||
| URL Spider Pro/x.xx (innerprise.net) | ||
| USyd-NLP-Spider | USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) | University of Sydney |
| Ultraseek | Ultraseek | Go |
| UnChaosBot | <b> UnChaosBot, From Chaos To Order, UnChaos Hybrid Web Search Engine at www.unc haos.com </b> (info@unchaos.com) |
UnCHAOS |
| <a href=”http://www.unchaos.com/”> UnChaos Bot, Hybrid Web Search Engine. </a> ( vadim_gonchar@unchaos.com) |
||
| <a href=”http://www.unchaos.com/”> UnChaos </a>, From Chaos To Order, Hybrid Web Search Engine. (vadim_gonchar@unchaos.com) |
||
| Uni-koblenz | http://www.uni-koblenz.de/~flocke/robot-info.txt | University of Koblenz |
| Unido-bot | unido-bot, http://mobicom.cs.uni-dortmund.de/bot.html | University of Dortmund |
| Updated | updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) | Updated |
| Updated crawler | updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com) | Updated |
| Uptimebot | Uptimebot | Uptimebot |
| UptimeBot(www.uptimebot.com) | ||
| Urlchecker1.0 | Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) | Euro Directory |
| Urlfan-bot | urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html | Urlfan |
| VBSEO | Mozilla/4.0 (vBSEO; http://www.vbseo.com) | VBSEO |
| VIProbot | VIPr/0.4 (VIProbot; http://www.vipsolutions.hu; info@vipsolutions.hu) | Vipsolutions |
| VMBot | VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) | VerticalMatch |
| VMBot/0.9 (VMBot; http://www.verticalmatch.com; vmbot@tradedot.com) | ||
| VWBOT | VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiu c.edu |
University of Illinois |
| Vacobot | Vacobot; (+http://vaco.ws/bot.html) | Vaco |
| Vagabondo | Vagabondo/3.0 (webagent at wise-guys dot nl) | WiseGuys |
| Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http ://webagent.wise-guys.nl/) |
||
| Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://w ebagent.wise-guys.nl/) |
||
| Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; htt p://webagent.wise-guys.nl/; http://www.wise-guys.nl/) |
||
| Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http:// webagent.wise-guys.nl/; http://www.wise-guys.nl/) |
||
| Vagabondo-WAP | Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/ )/1.0 Profile |
WiseGuys |
| Vakes | Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) | Vakes |
| Verizon | fast enterprise crawler 6 used by verizon superpages powered by fast (kevin.watt ers@fastsearch.com) |
Verizon Superpages |
| FAST Enterprise Crawler 6 used by Verizon Superpages Powered By FAST (crawler_ad min@superpages.com) |
||
| Vermut | Mozilla/5.0 (compatible; heritrix/@VERSION@ +http://vermut.aol.com) | AOL |
| mozilla/5.0 (compatible; vermut +http://www.aol.com) | ||
| Mozilla/5.0 (compatible; vermut +http://vermut.aol.com) | ||
| Versus | versus 0.2 (+http://versus.integis.ch) | Hochschule f?r Technik Rapperswil |
| Versus Crawler | versus crawler eda.baykan@epfl.ch | Ecole Polytechnique F?d?rale de Lausanne |
| VeryGoodSearch | VeryGoodSearch.com.DaddyLongLegs | VeryGoodSearch |
| Verzamelgids | verzamelgids.nl – Networking4all Bot/x.x | Verzamelgids |
| Verzamelgids/2.2 (http://www.verzamelgids.nl) | ||
| Vespa Crawler (Yahoo) | Vespa Crawler | Yahoo! Norway |
| Visbot | Visbot/1.0 (+http://www.visvo.com/bot.html;bot@visvo.com) | Visvo |
| VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) | ||
| Vlad | Mozilla/5.0 (compatible; vlad/1.1; +http://www.freshnotes.com/vlad.html) | Freshnotes |
| VoilaBot | KE_1.0/2.0 libwww/5.2.8 | Voila.fr |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (suppor t.voilabot@orange-ftgroup.com) |
||
| Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6 | ||
| Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (http:/ /www.voila.com/) |
||
| Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voi la.com/) |
||
| Vonna.com b o t | Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) | Vonna |
| Vortex/2.2 | Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) | Marty Anstey |
| Voyager | cfetch/1.0 | Kosmix |
| voyager-hc/1.0 | ||
| voyager/1.0 | ||
| Vspider | vspider | Verity |
| vspider/3.x | ||
| W3C-Validator | W3C_Validator/1.432.2.10 | W3C |
| W3C-checklink | W3C-checklink/4.2 [4.20] libwww-perl/5.803 | W3C |
| W3SiteSearch Crawler | W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de | W3SiteSearch |
| W8net | Mozilla/5.0 usww.com-Spider-for-w8.net | usww |
| WEP Search | WEP Search 00 | Unknown |
| WIRE | WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) | CWR |
| WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it) | ||
| WISEbot | WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) | Koreawisenut |
| WWW.fi crawler | www.fi crawler, contact crawler@www.fi | WWW.FI |
| WWWeasel Robot | WWWeasel Robot v1.00 (http://wwweasel.de) | World Wide Weasel |
| Wadaino.jp-crawler | wadaino.jp-crawler 0.2 (http://wadaino.jp/) | Wadaino |
| Wavefire | Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) | Wavefire |
| Waypath | Waypath development crawler – info at waypath dot com | Waypath |
| Waypath Scout v2.x – info at waypath dot com | ||
| WeRelateBot | WeRelateBot/0.9 (WeRelate; http://www.werelate.org/wiki/WeRelate:Bot; dallan@wer elate.org) |
WeRelate |
| WeatherBot | WeatherBot v1.4 http://www.ezweather.net | EZ Weather |
| WebAlta Crawler | WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) (Windows; U; Windows NT 5.1; ru-RU) |
WebAlta |
| WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows ; U; Windows NT 5.1; ru-RU) |
||
| WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html) | ||
| WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows ; U; Windows NT 5.1; ru-RU) |
||
| WebBOT | Carnegie_Mellon_University_WebCrawler,http://www.andrew.cmu.edu/~brgordon/webbot /index.html |
Carnegie Mellon University |
| WebCorp | WebCorp/1.0 | WebCorp |
| WebFindBot | WebFindBot(http://www.web-find.com) | web-find |
| WebGobbler | webGobbler/1.x.x | WebGobbler |
| WebRankSpider | WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) | WebRankSpider |
| WebSearch | WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; S earch@WebSearch.COM.AU) |
WebSearch |
| WebSearchBench | WebSearchBench WebCrawler v0.1(Experimental) | University of Dortmund |
| WSB, http://websearchbench.cs.uni-dortmund.de | ||
| WSB WebCrawler V1.0 (Beta), cl@cs.uni-dortmund.de | ||
| WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive rsität Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de / |
||
| WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Unive rsit?t Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de / |
||
| WebStat | WebStat/1.0 (Unix; beta; 20040314) | University of South Carolina |
| WebVac | WebVac (webmaster@pita.stanford.edu) | Stanford University |
| WebarooBot | WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) | Webaroo |
| WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html) | ||
| Webbandit | webbandit/4.xx.0 | SoftwareSolutions |
| Webbot | Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html) | Webbot |
| webbot(+http://webbot.com/bot.htm) | ||
| Webclipping | Webclipping.com | Webclipping |
| Webcrawl | webcrawl.net | Webcrawl |
| webcrawl.net | ||
| Webduniabot | Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) | Webdunia |
| Webglimpse | Webglimpse 2.xx.x (http://webglimpse.net) | Webglimpse |
| Weblog Attitude Diffusion | Weblog Attitude Diffusion 1.0 | Los Alamos National Laboratory |
| Webmeasurement-bot | webmeasurement-bot, http://rvs.informatik.uni-leipzig.de | University of Leipzig |
| Webmeasurement-crawler | webmeasurement-crawler, http://mobicom.cs.uni-dortmunde.de/webmeasurement/ | University of Dortmund |
| WebsiteWorth | WebsiteWorth v1.0 | Sootle |
| Webspinne | Webspinne/1.0 webmaster@webspinne.de | Webspinne |
| Websquash | Websquash.com (Add url robot) | Websquash |
| Webster | Webster v0.3 ( http://webster.healeys.net/ ) | Webster.healeys |
| Webverzeichnis | Webverzeichnis.de – Telefon: 01908 / 26005 | Webverzeichnis |
| Whoiam | Mozilla/5.0 whoiam [http://www.axxus.de/] | Axxus |
| Willow | Willow Internet Crawler by Twotrees V2.1 | Twotrees |
| WinME | Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] | ? |
| WinkBot | WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkB ot; winkbot@wink.com) |
Wink |
| WiseGuys | Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wi se-guys.nl/) |
WiseGuys |
| Vagabondo/2.0 MT | ||
| Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys. nl/) |
||
| Vagabondo/1.x MT (webagent@wise-guys.nl) | ||
| Vagabondo/2.0 MT (webagent at wise-guys dot nl) | ||
| Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl) | ||
| Worio | Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/) | Worio |
| Worio bot | Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com) | Worio |
| Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com) | ||
| Worio heritrix bot | worio heritrix bot (+http://worio.com/) | Worio |
| WorldWideWeb-X/3.1 | WorldWideWeb-X/3.1 (+http://www.worldwideweb-x.com/) | WorldWideWeb |
| Wotbox | Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 | Wotbox |
| Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com) | ||
| Wume_crawler | wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) | Lehigh University |
| Wwlib/Linux | Wwlib/Linux | Wolverhampton Web Library |
| Wwwster | wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de) | Centrum f?r Informations- und Sprachverarbeitung |
| XP5 | Project XP5 [2.03.07-111203] | Marty Anstey |
| Xdefine | egothor/3.0a (+http://www.xdefine.org/robot.html) | Xdefine |
| Xirq | xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) | Xirq |
| Xyleme SA France | cosmos/0.9_(robot@xyleme.com) | Xyleme |
| cosmos/0.8_(robot@xyleme.com) | ||
| Y!J | Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) | Yahoo Japan |
| Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) | ||
| Yacy | yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/ en) |
Yacy |
| Yacybot | yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net | Yacy |
| yacybot (i386 Linux 2.6.17-gentoo-r7; java 1.5.0_08; GMT/en) yacy.net | ||
| yacybot (i386 Linux 2.6.17-2-686; java 1.5.0_06; Europe/de) yacy.net | ||
| Yahoo Japan robot | DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear ch/indexing/indexing-27.html) |
Yahoo Japan |
| Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) | ||
| DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/sear ch/indexing/indexing-27.html) |
||
| Y!J-SRD/1.0 | ||
| Yahoo Search Japan robot | Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) | Yahoo Japan |
| Yahoo Search Marketing crawler | Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com) | Yahoo |
| YahooYSMcm/2.0.0 | ||
| Yahoo! Mindset | Yahoo! Mindset | Yahoo! Mindset |
| Yahoo-Blogs | Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/ us/ysearch/crawling/crawling-02.html ) |
Yahoo |
| Yahoo-MMAudVid | Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.co m ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0) |
Yahoo |
| Yahoo-MMCrawler | Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com) | Yahoo |
| Yahoo-Test/4.0 | Yahoo-Test/4.0 | Yahoo |
| Yahoo-VerticalCrawler | Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/ |
Yahoo |
| YahooFeedSeeker | YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.c om/rssguide) |
Yahoo |
| YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://my.yahoo.com/s/pu blishers.html) |
||
| YahooSeeker | YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-web master@yahoo-inc.com ) (KHTML, like Gecko) |
Yahoo |
| YahooSeeker/M1A1-R2D2 | SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-i nc dot com) |
Yahoo |
| LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile -search-customer-care AT yahoo-inc dot com) |
||
| MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible ;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling- 01.html) |
||
| SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL DC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-in c dot com) |
||
| Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration /CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;http://help.yahoo. com/help/us/ysearch/crawling/crawling-01.html) |
||
| SonyEricssonP910c/R2A SEMC-Browser/Symbian/3.0 Profile/MIDP-2.0 Configuration/CL DC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/ crawling/crawling-01.html) |
||
| SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com) |
||
| SIE-SX1/05 UP.Browser/7.0.0.1.181 (GUI) MMP/2.0 Profile/MIDP-2.0 Configuration/C LDC-1.1 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch /crawling/crawling-01.html) |
||
| Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration /CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo .com/help/us/ysearch/crawling/crawling-01.html) |
||
| Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration /CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-cust omer-care AT yahoo-inc dot com) |
||
| SGH-Z130 SHP/VPP/R5 SMB3.1 SMM-MMS/1.1.0 profile/MIDP-2.0 configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawli ng/crawling-01.html) |
||
| LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http: //help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) |
||
| Nokia9500 (compatible;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo -inc dot com) |
||
| MOT-V975/81.33.02I MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 (compatible ;YahooSeeker/M1A1-R2D2;mobile-search-customer-care AT yahoo-inc dot com) |
||
| Yandex | Mozilla/4.0 (compatible; MSIE 5.0; YANDEX) | Yandex |
| Yandex/1.01.001 (compatible; Win16; I) | ||
| Yarienavoir | yarienavoir.net/0.2 | Yarienavoir |
| Yellopet | spider.yellopet.com – www.yellopet.com | Yellopet |
| Yeti | Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) | Naver |
| Yeti | ||
| Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it) | ||
| Yggdrasil | yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu m inus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dres den dot de) |
GoPubMed |
| YodaoBot | Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spide r/; ) |
Yodao |
| YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; ) | ||
| YooW! | YooW!/1.8.9 RC1 (+http://www.yoow.eu) | YooW! |
| Yoogli | yoogliFetchAgent/0.1 | Yoogli |
| Yoono | Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) | Yoono |
| Yoono web-crawler | yoono/1.0 web-crawler/1.0 | Yoono |
| YottaCars | YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine | YottaCars |
| YottaShopping | YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine | YottaShopping |
| Z-Add Link Checker | Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) | Z-add |
| Zao Crawler | Zao-Crawler | Kototoi |
| Zao-Crawler 0.2b | ||
| Zao/0.1 (http://www.kototoi.org/zao/) | ||
| ZeBot | ZeBot_www.ze.bz (ze.bz@hotmail.com) | ZE.BZ |
| ZeBot_lseek.net (bot@ze.bz) | ||
| Zealbot | Mozilla/4.0 (compatible; Zealbot 1.0) | LookSmart |
| Zearchit | Zearchit | Zearchit |
| Zedzo.digest | zedzo.digest/0.1 (http://www.zedzo.com/) | Zedzo |
| Zerxbot | zerxbot/Version 0.6 libwww-perl/5.79 | Zerx |
| Zetbot | DeepIndex ( http://www.zetbot.com ) | Zetbot |
| Zeus | ZBot/1.00 (icaulfield@zeus.com) | |
| Zeus xxxxx Webster Pro V2.9 Win32 | ||
| Zeus ThemeSite Viewer Webster Pro V2.9 Win32 | ||
| ZipppBot | ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net) | Zipp |
| ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) | ||
| Zippy | Zippy v2.0 – Zippyfinder.com | Zippyfinder |
| ZoomSpider | ZoomSpider – wrensoft.com | WrenSoft |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR 1.1.4322) |
||
| Zspider | zspider/0.9-dev http://feedback.redkolibri.com/ | Redkolibri |
| ZyBorg (LookSmart) | ZyBorg | LookSmart |
| ZyBorg (Wisenut) | Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnu tbot.com) |
Wisenut |
| ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) |
全球搜索引擎最新蜘蛛_爬虫列表
版权归原作者所有，如有侵权请告知。达维营-前端网 » 全球搜索引擎最新蜘蛛_爬虫列表