# skeeterlab.com # # NOTE ON /impressum.html AND /privacy.html # # Those two pages are deliberately NOT disallowed here, even though we do not # want them indexed. This is counterintuitive and it is the single most common # way people get this wrong: # # Disallow stops the crawler FETCHING the page. A crawler that never fetches # the page never sees its noindex directive, so Google can still list the # bare URL with "No information is available for this page" — the page still # surfaces under the domain. Blocking makes the problem permanent. # # The removal instruction has to be readable, so the crawl must be allowed. # noindex is enforced in two places instead: # 1. X-Robots-Tag response header — infra/noindex-legal.js (CloudFront) # 2. — in the pages themselves # # Use this file only to shed crawlers we do not want at all. User-agent: * Allow: / # --------------------------------------------------------------------------- # AI / model-training crawlers — declined. # --------------------------------------------------------------------------- User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: omgili Disallow: / # --------------------------------------------------------------------------- # Web archives — kept off the legal pages only. # # noindex/noarchive are search-engine directives; the Internet Archive does not # act on them. Without this, the Wayback Machine snapshots the Impressum and # keeps the name and address reachable forever, outside our control and immune # to any later takedown on our side. # # Disallowing here is safe and does NOT repeat the trap described above: that # trap is about search indexers listing a bare URL they were never allowed to # read. Archive crawlers produce no search results, so blocking the fetch is # exactly the outcome we want. The rest of the site stays archivable. # --------------------------------------------------------------------------- User-agent: ia_archiver Disallow: /impressum.html Disallow: /impressum Disallow: /privacy.html Disallow: /privacy Disallow: /datenschutz.html Disallow: /datenschutz Disallow: /de/privacy.html Disallow: /fr/privacy.html Disallow: /es/privacy.html User-agent: archive.org_bot Disallow: /impressum.html Disallow: /impressum Disallow: /privacy.html Disallow: /privacy Disallow: /datenschutz.html Disallow: /datenschutz Disallow: /de/privacy.html Disallow: /fr/privacy.html Disallow: /es/privacy.html Sitemap: https://skeeterlab.com/sitemap.xml