npm packagerobots-parser (npm package)

Robots.txt parser.


GitHub repos using robots-parser (showing top 100 with most stars)

#GitHub repo nameStars
1GoogleChrome/lighthouse17k
2yujiosaka/headless-chrome-crawler4.2k
3simplecrawler/simplecrawler1.9k
4brendonboshell/supercrawler200+
5ampproject/ampbench50+
6carlos8f/zenbrain30+
7willianjusten/crawler-huge9
8fractalf/edderkopp8
9paulirish/lighthouse7
10passmarked/seo7
11LewisTehMinerz/spiderboi4
12zeit/lighthouse3
13nunnun/supercrawler3
14GeorgeA93/crawley3
15tobiashinz/remote-sitemap-generator2
16gajus/robots-agent2
17yukiii-zhong/lighthouse1
18ymm8505/lighthouse1
19yagoubigithub/lighthouse1
20yWorks/simplecrawler1
21vinuthaharishankar/lighthouse1
22vbman2012/lighthouse1
23tuffchip/lighthouse1
24tamermaarouf/lighthouse1
25tafaquh/lighthouse1
26sylturner/headless-chrome-crawler1
27syavash/simplecrawler1
28songpx/lighthouse1
29shong13/lighthouse1
30shellsec/headless-chrome-crawler1
31sh1583/lighthouse1
32sensetv/lighthouse1
33saidhaoua/lighthouse1
34plugnsearch/plugnsearch1
35philkrie/ampbench1
36or73/lighthouse1
37olamiwhat/lighthouse1
38nimondo/lighthouse1
39niilante/lighthouse1
40nayramo/lighthouse1
41nadrane/crawler1
42mustafaelnagar/lighthouse1
43marvelperseus/lighthouse1
44manueljlora/lighthouse1
45labolado/headless-chrome-crawler1
46kesarlong/lighthouse1
47joeyparrish/lighthouse1
48jnaghdi/lighthouse1
49indigos33k3r/headless-chrome-crawler1
50imagineagents/lighthouse1
51herberthk/lighthouse1
52harshit98/lighthouse1
53haoyunfeix/lighthouse1
54geekwise-alonzo-yrigollen/lighthouse1
55gautamsharma0095/supercrawler1
56eduardotp77/lighthouse1
57dylanb/lighthouse1
58cindyjetskis/lighthouse1
59brhmbrgl/lighthouse1
60bobkali/lighthouse1
61andreibad/lighthouse1
62alejagapatrick/lighthouse1
63akojif/lighthouse1
64agencyanalytics/act-crawler1
65adnanmasoud/lighthouse1
66ZehuaWang/headless-chrome-crawler1
67Yassinestaili/lighthouse1
68YashPrajapati320/lighthouse1
69Yairimedinac4621/lighthouse1
70Wyte3z/lighthouse1
71Shaggy96669/lighthouse1
72RipleyLudicrous1979/node-simplecrawler1
73ReckerPhildev/lighthouse1
74RadouaneBounefla/lighthouse1
75QuantumCloudDatabase/lighthouse1
76Prisguiden/edderkopp1
77PeeYa/lighthouse1
78PROwassim/lighthouse1
79Nipher/crawler-huge1
80MohamedHesham/lighthouse1
81MindyBrice1969/lighthouse1
82L-fa/lighthouse1
83Kylekoh/lighthouse1
84Engyunlimited/lighthouse1
85DalavanCloud/ampbench1
86ColleenKeegan/lighthouse1
87COOLDUDE5700/lighthouse1
88BrankoVukmirovic/lighthouse1
89BenPeach1/lighthouse1
90Azier777/lighthouse1
91ArCate-Sun/lighthouse1
92Amal-hossam/lighthouse1
93ARYANKAUSHIK/seo1