User-agent: * Disallow: /*?* Disallow: /*.zip$ Disallow: /updates/* #stole from sr.ht # Too aggressive, marketing/SEO User-agent: SemrushBot Disallow: / # Too aggressive, marketing/SEO User-agent: SemrushBot-SA Disallow: / # Huwei something or another, badly behaved User-agent: AspiegelBot Disallow: / # too aggressive User-agent: Yandex Disallow: / # No User-agent: turnitinbot Disallow: / # Does not respect * directives User-agent: Seekport Crawler Disallow: /