flounder

A simple gemini site builder
git clone git://git.alexwennerberg.com/flounder
Log | Files | Refs | README | LICENSE

robots.txt (451B) - raw

      1 User-agent: *
      2 Disallow: /*?*
      3 Disallow: /*.zip$
      4 
      5 #stole from sr.ht
      6 
      7 # Too aggressive, marketing/SEO
      8 User-agent: SemrushBot
      9 Disallow: /
     10 
     11 # Too aggressive, marketing/SEO
     12 User-agent: SemrushBot-SA
     13 Disallow: /
     14 
     15 # Huwei something or another, badly behaved
     16 User-agent: AspiegelBot
     17 Disallow: /
     18 
     19 # too aggressive
     20 User-agent: Yandex
     21 Disallow: /
     22 
     23 # No
     24 User-agent: turnitinbot
     25 Disallow: /
     26 
     27 # Does not respect * directives
     28 User-agent: Seekport Crawler
     29 Disallow: /