flounder

A simple gemini site builder
git clone git://git.alexwennerberg.com/flounder
Log | Files | Refs | README | LICENSE

commit cced693f359dea3437e34bce00275613719e95e0
parent 03f5abe6f484ba8e0bb9ae0aba44104e8f2ad86d
Author: alex wennerberg <alex@alexwennerberg.com>
Date:   Fri, 28 Oct 2022 22:47:36 -0700

add robots.txt

Diffstat:
Mtemplates/static/robots.txt | 26++++++++++++++++++++++++++
1 file changed, 26 insertions(+), 0 deletions(-)

diff --git a/templates/static/robots.txt b/templates/static/robots.txt @@ -1,3 +1,29 @@ User-agent: * Disallow: /*?* Disallow: /*.zip$ + +#stole from sr.ht + +# Too aggressive, marketing/SEO +User-agent: SemrushBot +Disallow: / + +# Too aggressive, marketing/SEO +User-agent: SemrushBot-SA +Disallow: / + +# Huwei something or another, badly behaved +User-agent: AspiegelBot +Disallow: / + +# too aggressive +User-agent: Yandex +Disallow: / + +# No +User-agent: turnitinbot +Disallow: / + +# Does not respect * directives +User-agent: Seekport Crawler +Disallow: /