flounder

A simple gemini site builder
git clone git://git.alexwennerberg.com/flounder
Log | Files | Refs | README | LICENSE

robots.txt (472B) - raw

      1 User-agent: *
      2 Disallow: /*?*
      3 Disallow: /*.zip$
      4 Disallow: /updates/*
      5 
      6 #stole from sr.ht
      7 
      8 # Too aggressive, marketing/SEO
      9 User-agent: SemrushBot
     10 Disallow: /
     11 
     12 # Too aggressive, marketing/SEO
     13 User-agent: SemrushBot-SA
     14 Disallow: /
     15 
     16 # Huwei something or another, badly behaved
     17 User-agent: AspiegelBot
     18 Disallow: /
     19 
     20 # too aggressive
     21 User-agent: Yandex
     22 Disallow: /
     23 
     24 # No
     25 User-agent: turnitinbot
     26 Disallow: /
     27 
     28 # Does not respect * directives
     29 User-agent: Seekport Crawler
     30 Disallow: /