robots.txt (451B) - raw
1 User-agent: * 2 Disallow: /*?* 3 Disallow: /*.zip$ 4 5 #stole from sr.ht 6 7 # Too aggressive, marketing/SEO 8 User-agent: SemrushBot 9 Disallow: / 10 11 # Too aggressive, marketing/SEO 12 User-agent: SemrushBot-SA 13 Disallow: / 14 15 # Huwei something or another, badly behaved 16 User-agent: AspiegelBot 17 Disallow: / 18 19 # too aggressive 20 User-agent: Yandex 21 Disallow: / 22 23 # No 24 User-agent: turnitinbot 25 Disallow: / 26 27 # Does not respect * directives 28 User-agent: Seekport Crawler 29 Disallow: /