npm packagecrawler (npm package)

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously


GitHub repos using crawler (showing top 100 with most stars)

#GitHub repo nameStars
1f111fei/article_spider100+
2zhang435/Music-Porter70+
3jfhbrook/hoarders50+
4Live-API/LAPI-Server40+
5xidui/algorithm-training40+
6mesosphere/dcos-docs-site40+
7i5ting/githubrank20+
8i5ting/simplereader20+
9ntblk/block-crawler10+
10wepyjs/weapp-document-tracker10+
11nullpointer/fund-crawler10+
12moein7tl/email-extractor10+
13devfsa/remotocc-crawler10+
14bumbeishvili/fb-geo-jobs10+
15sarthology/criCLI10+
16matthewhudson/snailer10+
17gdi2290/node-everything10+
18aisinvon/ThinkJS-Demo10+
19raphaelkieling/PodNews10+
20mikedeng/city_geojson10+
21Robinsir/BTSOW10+
22luxifer/earthview-scrap10+
23five6/node_vue_react10+
24shoutrlabs/berliner-schlagzeilen7
25mrzhuzhe/punisher7
26luicfer/images-spider7
27lanzhiheng/jianshu-crawler7
28christianvuerings/berkeleydir7
29HackManiac/Ruhestein7
30Bersam/radiojavancrawler7
31ruffrey/google-finance-crawler6
32johansatge/psi-report6
33hh54188/node-proxy-pool6
34haxzie/instagram-downloader6
35grv19/livescores6
36ftto/peoplepower-crawler-people6
37PhilipDing/linux6
38Lyule/node-crawler-region6
39Cookizza/FAA-Chart-Downloader6
40wilsonandusa/GitHub-Hunter5
41washingtonstateuniversity/WSU-Web-Crawler5
42metux/nodejs-instagram5
43kevinLiJ/NBARealTimeScore5
44jimmymorris/birddog-cli5
45funnycoderstar/yys-be5
46buzzybeeio/jobsearch5
47blaryjp/lighthouse-report5
48MoritzGiessmann/naiveScript5
4918569505972/spider-book5
50zj-john/autoCommit4
51woaiso/wst4
52tjjh89017/2dg4
53fund-ui/fu-crawler4
54filet-mign0n/countrygraph4
55chrismatthieu/crawler4
56Huoshendameng/Node-Interface4
57Godoy/node-imdb-crawler4
58FelixLinBH/twExchange4
59wilsonandusa/DoctorWat3
60techvn/crawler3
61storyblok/lambda-crawler-aws-elasticsearch3
62sogko/node-ckan-crawler3
63semisleep/simple-licensing-api3
64kaytcat/slack-lunchbot3
65jbnicolai/urlgraph3
66i5ting/hd-crawler3
67haxorpoda/bug-cruncher3
68grissius/offensive-fortune3
69datagovsg/credits-generator3
70codeforsanjose/calidrought-legacy3
71anvaka/allcomposer3
72ZZR-china/super-crawler3
73Sirikon/astenagusia3
74Hugoer/filmaffinityParser3
75Emersonbrandon0/searchtermcrawler3
76Devs-Garden/WebCrawler3
77CoinQuant/btcbot3
78AravindRajeshkanna/arkrawler3
79451hackathon/block-crawler3
8010iii/getcd-org3
81zuiidea/sube-api2
82zhoujinhai/nightmare-crawler-youtube2
83zhanziyang/billboard-crawler2
84yuji8023/crawler2
85yinxin630/hackthon2
86xinchanghao/jsDemo2
87xiaonan132142/gc-server2
88xb1itz/muz.li2
89withinjs/hotspot2
90williamdes/mariadb-mysql-kbs2
91storyblok/lambda-crawler2
92sjy/node_scrapy2
93silentrob/superscript-friends2
94sanli/cag-imgtools2
95qiangshuifish/node-spider2
96picturepan2/xboxlivestore2
97onyxcorp/scrapper2
98ockang/cine.com2
99node-napkin/napkin2
100niksmac/mitd-crawler2
101monavari-lebrecht/OpenUI5Docs2
102merrier/overwatch-spider2
103linoleum-js/salary-crawler2
104justinsilvestre/classical-chinese2
105joeeasy/AlgorithmAnalysis2
106jcblw/gimmeztehcatz2
107goveo/ticker-parser2
108ftto/peoplepower-crawler-laws2
109ffis/sici2
110fabrix-app/spool-scraper2
111eces/ecesly.com2
112duyetdev/DCrawler2
113dev-dude/automatic-website-builder2
114btc022003/book-shop2
115boboidream/article_spider2
116anvaka/allbower2
117YanceyOfficial/cute-rubbish2
118Sicau-HsuYang/node-crawl-demo2
119RongJunyu/Tianya_Spider2
120Nexters/clip_web2
121LouisTakePILLz/node-visa-scraper2
122HellowVirgil/fiction-crawler2
123FranciscoKnebel/osrs-latency2
124CodersCrew/inStudy2
125AppliedSoul/promise-crawler2
12652cik/kindle-spider-xiaohua2
127zycbobby/easy_download1
128zj-john/houseCrawler1
129zitang/idiom_crawler1
130zhuruliang/novel-crawler1
131zhoushirong/crawler1
132zfowed/myapp1
133zengibm/webpack1
134zcg331793187/DownloadYouLike1
135z-hao-wang/crawler-hylia1
136yuetsh/hackathon1
137xu-jian-bin/x-crawler1
138xinhuaRadioLAB/learn-node-crawler1
139wustcyl/giligili1
140wenkangjing/txt-crawler1
141webholics/node-trader1
142victoryancn/fm-crawler1
143victorpinheiro/recipes-crawler1
144vanhelsinglala/algorithm-training1
145useful-cli/backend-cli1
146topshadow/spring-wind1
147tomoat/anccnet-spider1
148thepearson/algolia-site-crawler1
149thanhbka89/nodejs_demo1
150tbarnes94/fortnite-weapons-bot1
151sunjoong85/istock1
152stol/poligame1
153soakit/book-download1
154shock-lee/express-bilibili1
155shirts/gimmeztehcatz1
156seancheung/magicrawler1
157salielim/seoroberto1
158rostok/cdadl1
159roderickhsiao/So-You-Think-You-Can-Dance1
160ricardofbarros/icon-crawler1
161rainmakeross/kisearch-crawler1
162pkuimyy/cnki-user-info1
163patrickzxk/spider1
164parkmee/gift-fetcher1
165panhuibingithub/NoteCrawler1
166oshalygin/biaja-bookings1
167nswbmw/article_spider1
168nothing2lose/dictionary-crawler1
169norbikaa/nCore-scraper1
170node-migrator-bot/hoarders1
171nitin1998/ToiCrawler1
172navs23/mykathua.com1
173namuol/ludum-crawler1
174mvanmeerbeck/searchtrail1
175muhshahabipour/test-031
176msultan96/CS355-Search-Engine1
177mrbigs/spool-scraper1
178mojanr/alahawaw1
179mofumofu3n/hatena_crawler1
180mikestopcontinues/mikestopcontinues.com1
181michalhepner/webduck1
182melvin0008/crawl-emailids1
183meidawei/Node-crawler-ali1
184mboufos/docs1
185mateogo/develar1
186marceloabk/Magic-Price1
187mailtokamalnayan/apple-stories1
188mactive/btcserver1
189mabahmani/amazonc1
190logicalangel/amazonc1
191logicalangel/ImdbCrawler1
192littleVege/nodeAjkCrawler1
193langzitc/tuchun1
194klasikcorp30/comp3161-project1
195kimi0230/esunForex1
196kevinw/everylastmodel1
197kbenrafik/scrap1
198joshmcarthur/dora1
199joahCN/dinner1
200jlozovei/link-checker1
201jasonwu0731/PaperNet1
202jamesohn/Youtube-project1
203hello2dj/rss-crawler1
204haydenjameslee/Top-of-Reddit1
205handcrafter/secret-admirer1
206guillaumeboniface/sedlex1
207gowrav/imdb_parser1
208godong9/nodejs-boilerplate1
209gastrodia/carfans-api1
210fuchao2012/zhoukan1
211fike/dcos-docs-site1
212esraaghanemali/GE-Web-Scraper1
213eruixma/trading-app1
214ekevjn/shoper1
215dxdleikai/node_crawler1
216duyetdev/Crawler-DKXT-DH-20151
217duonghung1269/slackbot-toto-4d-singapore1
218dirutix/ticker-parser1
219direvius/site-analyzer1
220decodez/cosmic_crawler1
221cvdat2097/node-crawler1
222cupyong/crawl1
223crazydev71/coinmarketcap-scrapper1
224cjy37/article_spider1
225chuson1996/speed-vocab1
226christianvuerings/youtuby1
227chenyunhb/blog1
228cheneddie/eddie-node-crawler1
229caoyonghao/siteAnalysisTool1
230canercak/posluga1
231candreoliveira/crawler1
232builtinnya/mediaarts-db-crawler1
233budactw/novels-hunter-js1
234brunoventura/brejas-crawler1
235brunocalixto/crawler1
236bluefriday/ESC_DELight1
237bigmeech/mangaka1
238banminkyoz/scripts1
239banminkyoz/all-repos1
240backbrace/dts-generator1
241atefBB/nodecrawler1
242asalonen/reptilian1
243anvaka/allcran1
244alissoncs/busaosl1
245ajpyoung/UpWork_EntranceExam1
246ahangchen/article_spider1
247ZhangWei-UUU/express-next-react1
248YuukanOO/hubot-adventure1
249YanceyOfficial/learn-react1
250UCDavisLibrary/library-site-crawler1
251TomIsion/ONE-Vue1
252TNT-Likely/crawler1
253TEODE/domain-name-finder1
254ShaunDeng/dcos-docs-site1
255Samuel-Rochette/Aws_Scraping_Script1
256SalahudinMalik/nodejs_crawler1
257RogerZZZZZ/lexicon-cli1
258Rafael-Martins/API-Jogo-do-Bicho1
259Munkkeli/PriceCrawler1
260MohammadFakhreddin/forum-search-engine1
261MelvinTo/csm_crawler1
262MarcoWorms/steam-neural1
263Jinjeok/boty1
264IbrahimSassi/eReputation_ATS1
265Hfimy/wxCrawler1
266HXWfromDJTU/flower-manager1
267FranklinCncr/GT---Guia-de-Turismo1
268Dujthep/crawler1
269Diwatt/HeroesStatistics1
270Dafrok/awesome-docker-libraries1
271Cooper111/article_spider1
272CoinQuant/coin-news1
273ChandlerVer5/puppeteer-crawl1
274Chandler-Sun/wechat-user-bot1
275BeijingProtoHuman/douyin-location1
276Basel-Issmail/job-crawler1
277AutumnFish/lolData1
278AppliedSoul/crawlmatic1
279AntonyHatchet/FiShop1
280755966092/node-crawler-yiche1
281335296558/koa-mvc1
28216520511/Inventory-Updater-Manager1