# ============================================================ # robots.txt for getxeo.ai # Spec: https://www.rfc-editor.org/rfc/rfc9309 # Sitemaps: https://www.sitemaps.org/protocol.html # Generated: 2026-09-15 (rebuilt every Hugo deploy) # # ------------------------------------------------------------ # Crawl strategy: two goals, two different levers # ------------------------------------------------------------ # 1. Keep pages OUT of search results (Google, Bing, ...) # -> use a page-level meta robots "noindex". # -> the crawler MUST be allowed to crawl the page, or it # can never read the noindex. A robots.txt Disallow is # COUNTERPRODUCTIVE here: it blocks the crawl, the noindex # is never seen, and the URL can still be indexed as a # bare URL ("Indexed, though blocked by robots.txt"). # # 2. Keep content OUT of AI training/caching corpora # -> use a robots.txt Disallow for those crawlers. # -> they respect Disallow but do NOT honor a meta "noindex" # (noindex is a search-index directive, meaningless to a # training crawler). # # Because the two goals need opposite treatment for the SAME URLs, # the rules below are split by crawler type: # # - Search engines ........ full crawl access. Page-level noindex # (set in data/metadata/*.json) removes # policy + thin pages from the index # cleanly: no snippet, no cache. # - AI caching/training ... blocked from policy + thin pages so # legal text is not ingested or served # as a stale paraphrase. Everything else # stays open (AEO-first posture). # - AI live/browsing ...... allowed everywhere, including policy # pages. These fetch on demand for a # specific user query, so answers reflect # the CURRENT text (no staleness risk). # - Bytespider ............ blanket block (documented rate-limit # abuse + no AEO/GEO value for us). # # IMPORTANT robots.txt semantics: # A crawler obeys ONLY the single most-specific User-agent group # that matches its name and IGNORES every other group, including # "*". So each bot's full ruleset must live inside its own group. # "Allow: /" (or an empty "Disallow:") means "allow everything". # ============================================================ # ------------------------------------------------------------ # SEARCH ENGINES - full crawl access # ------------------------------------------------------------ # Policy pages (privacy/refund/shipment/terms) and /contact-sales/ # are all noindex at the page level. They are # intentionally NOT disallowed here: search bots must crawl them to # see the noindex, which is what actually keeps them out of the # index. Disallowing here would re-create the URL-only indexing bug. # # /api/ is the one exception. Those paths are backend form endpoints, # not pages: they answer POST and have no crawlable representation. # Search Console picked up /api/site-audit/start as an indexing # candidate, which is a URL a crawler should never have been offered in # the first place. Disallowing the prefix keeps the whole namespace out # of discovery. It does not touch /site-audit//, which is a # different path and must keep resolving for shared result links. # # /sitemap-others.xml is disallowed here as well. It holds the features # catalogue, /about/ and the product guide, and is not referenced from # sitemap.xml, so nothing in it is submitted. Note this rule is repeated # in the default group below AND set here: a crawler obeys only its own # most specific group, so the equivalent sitemap-archives.xml rule in the # default group alone never applied to the search engines named here. # Disallowing a sitemap file stops the file being fetched. It does not # deindex the pages listed inside it, which keep their own page level # meta robots and their internal links. # /industry-reports/ is deliberately absent from every Disallow in this # file. Those pages are original research and the whole point is that # search engines and AI engines both find and cite them, so they stay # open to every group below except Bytespider, which is blocked wholesale # for reasons unrelated to content. sitemap-industry-reports.xml is # likewise NOT disallowed, unlike sitemap-others.xml above. User-agent: Googlebot User-agent: Bingbot User-agent: DuckDuckBot User-agent: Slurp User-agent: Yandex User-agent: Baiduspider Disallow: /api/ Disallow: /sitemap-others.xml Allow: / # ------------------------------------------------------------ # AI CACHING / TRAINING CRAWLERS - block sensitive + thin pages # ------------------------------------------------------------ # These build persistent training corpora or caches, so blocking the # policy pages prevents stale paraphrases of legal text. All other # pages remain open to them. # GPTBot OpenAI training crawler # OAI-SearchBot ChatGPT Search index crawler (caches) # ClaudeBot Anthropic training crawler # anthropic-ai legacy Anthropic crawler # PerplexityBot Perplexity index crawler (caches) # Google-Extended Google AI (Gemini/Vertex) training # Applebot-Extended Apple Intelligence training # CCBot Common Crawl (upstream for many models) User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: /privacy-policy/ Disallow: /refund-policy/ Disallow: /shipment-policy/ Disallow: /terms-and-conditions/ Disallow: /contact-sales/ # ------------------------------------------------------------ # AI LIVE / BROWSING AGENTS - full access # ------------------------------------------------------------ # These fetch a page in real time to answer a specific user prompt, # so they always read the current text. Allowed everywhere so that # "what is getxeo's refund policy?" gets an accurate, live answer. # If a future policy review decides live AI summaries of legal text # are also too risky, move these three agents into the AI caching/ # training group above. # ChatGPT-User ChatGPT live browsing # Claude-Web Claude live browsing # Perplexity-User Perplexity live browsing User-agent: ChatGPT-User User-agent: Claude-Web User-agent: Perplexity-User Allow: / # ------------------------------------------------------------ # DEFAULT GROUP - every other crawler # ------------------------------------------------------------ # Unknown/unlisted bots: block the policy + thin pages by default. User-agent: * Disallow: /api/ Disallow: /privacy-policy/ Disallow: /refund-policy/ Disallow: /shipment-policy/ Disallow: /terms-and-conditions/ Disallow: /contact-sales/ Disallow: /sitemap-archives.xml Disallow: /sitemap-others.xml # ------------------------------------------------------------ # BLOCKED ENTIRELY - known abusive scraper # ------------------------------------------------------------ # Bytespider (ByteDance/TikTok): documented history of ignoring # rate-limit headers and crawling at DoS-like rates; no discovery # value for our category. User-agent: Bytespider Disallow: / # ------------------------------------------------------------ # ARCHIVED BLOG CORPUS # ------------------------------------------------------------ # The 106 posts under /blogs/ were archived. They are noindex at the # page level (page.robots_directives in each # data/metadata/blogs-.json) and are deliberately NOT disallowed # below: a blocked crawler never reads the noindex, and the URL can # still surface as a bare listing. Same reasoning as the policy pages # above. Crawl stays open so the noindex is seen and obeyed. # # Only the archives SITEMAP FILE is disallowed, and that rule lives in # the default group below rather than in a second wildcard group: two # "User-agent: *" groups in one file is ambiguous, since a crawler is # meant to obey a single matching group. Disallowing a sitemap file # does not affect crawling of the pages it lists. # ------------------------------------------------------------ # Sitemap discovery # ------------------------------------------------------------ # Submit ONLY this index in GSC. It references only sub-sitemaps that # contain indexable URLs (sitemap-core for marketing + feature detail # pages, sitemap-blogs which is currently empty and reserved for the # new blog corpus). The archived posts in sitemap-archives.xml are # noindex, so that file is not referenced by the index at all -- a # sitemap entry says "index me" and contradicts a noindex. Sitemap: https://getxeo.ai/sitemap.xml