robots.txt (472B) - raw
1 User-agent: * 2 Disallow: /*?* 3 Disallow: /*.zip$ 4 Disallow: /updates/* 5 6 #stole from sr.ht 7 8 # Too aggressive, marketing/SEO 9 User-agent: SemrushBot 10 Disallow: / 11 12 # Too aggressive, marketing/SEO 13 User-agent: SemrushBot-SA 14 Disallow: / 15 16 # Huwei something or another, badly behaved 17 User-agent: AspiegelBot 18 Disallow: / 19 20 # too aggressive 21 User-agent: Yandex 22 Disallow: / 23 24 # No 25 User-agent: turnitinbot 26 Disallow: / 27 28 # Does not respect * directives 29 User-agent: Seekport Crawler 30 Disallow: /