# councilof.ai — Council of AI (CSOAI LTD, UK 16939677) # # CORRECTED 6 August 2026. The previous file contained TWO `User-agent: *` groups and # listed ClaudeBot, GPTBot and Google-Extended twice each with opposite instructions — # Disallow: / in the first group, Allow: / in the second. Under RFC 9309 a crawler selects # the most specific matching group; where two groups are equally specific the outcome is # undefined and implementations differ. Nobody could predict whether those three crawlers # read this site, including the crawlers. # # This file states one thing per agent. Every agent appears exactly once. # --------------------------------------------------------------------------- # Content signals. Retained from the previous file — these were correct. # # Per the Content-Signal specification: if a signal is `yes` you may collect content for # that use; if `no` you may not; if a signal is absent, permission is neither granted nor # restricted by this mechanism. # # search — building a search index and returning links and short excerpts. # Does not include AI-generated search summaries. # ai-input — inputting content into AI models at request time: RAG, grounding, # generative search answers. # ai-train — training or fine-tuning AI models. # use — how AI systems may consume the content: immediate, reference, or full. # # THE `ai-train=no` SIGNAL IS AN EXPRESS RESERVATION OF RIGHTS under Article 4(3) of # Directive (EU) 2019/790 and is intended to have that legal effect. It is deliberate and # should not be removed without a decision to license training use. # # `ai-input` is set to `yes`. This is the change of substance in this file. This estate # publishes measurements so that they are read, cited and checked; an answer engine # grounding a response in a CSOAI measurement is the intended use, and it is a different # act from training on it. Reserving training while permitting grounding is a coherent # position and is exactly the one the site's copy describes. # --------------------------------------------------------------------------- # Feeds and paste-ready badges. The feed is the estate's only return path; it was # previously reachable only at /api/feed.xml, which nothing guesses. # Feed: https://councilof.ai/feed.xml (alias /rss.xml, canonical /api/feed.xml) # Badges: https://councilof.ai/badge.md (plain markdown, every snippet in the body) User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: /.backups/ Disallow: /_backup/ Disallow: /api/ # The living board is public evidence and is listed in the sitemap. Keep the # rest of /api/ excluded while allowing crawlers to read the board and its # documented query-string filters. Allow: /api/gspc # AGENT-FACING ENDPOINTS — ALLOWED (26 September 2026). `Disallow: /api/` above also # blocked the endpoints this estate publishes FOR agents: the x402 catalogue, the A2A # interface, the MCP catalogue, the self-description (/api/state), the OpenAPI document, # the signed card store and the public root. A well-behaved agent that honours robots.txt # was told to stay away from exactly the doors it was invited to. Under RFC 9309 the # longest matching rule wins, so each Allow below overrides the shorter Disallow for its # own path; the rest of /api/ stays excluded. The Content-Signal above is unchanged: # ai-train=no still applies to everything these paths serve. Allow: /api/x402 Allow: /api/a2a Allow: /api/mcp Allow: /api/state Allow: /api/openapi.json Allow: /api/cards Allow: /api/root # The canonical RSS feed (named at the top of this file) is listed in the sitemap, but # `Disallow: /api/` told crawlers not to fetch it at all (found 2026-09-28). Allow: /api/feed.xml # Cloudflare rewrites every e-mail address into a /cdn-cgi/l/email-protection link that answers # 404 to a crawler; on 2026-09-28 that dead link sat on 303 of the 443 sitemap pages. Disallow: /cdn-cgi/ Disallow: /*?*utm_ Allow: / # The same Content-Signal line is repeated in EVERY named group below (26 September 2026). # Under RFC 9309 a crawler that finds a group naming it reads ONLY that group and never # falls back to `*`. So GPTBot, ClaudeBot, CCBot and the rest, each with a group of its own, # never saw the `ai-train=no` reservation stated in the `*` group. The Allow and Disallow # lines of every group are unchanged; only the signal is added. The test in # functions/api/robots-agent-paths.test.ts fails if any group is missing it. # --------------------------------------------------------------------------- # Answer engines and AI search crawlers — ALLOWED. # # These fetch pages to ground and cite answers. Blocking them removes this estate from # the surface where its buyers now ask their first question, and it does not prevent # training: training use is reserved by the Content-Signal above and by the terms, which # is the mechanism designed for it. A robots block is a discovery decision, not a # copyright one, and blocking these agents was costing discovery for no gain. # # Each appears exactly once. No agent below is repeated anywhere in this file. # --------------------------------------------------------------------------- User-agent: Claude-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: Claude-User Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: Applebot-Extended Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / User-agent: meta-externalagent Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Allow: / # --------------------------------------------------------------------------- # Bulk scrape corpora — DISALLOWED. # # These assemble general-purpose training corpora rather than answering a user's question # about this site. Refusing them is consistent with `ai-train=no` and costs no discovery, # because nothing here is cited back to a reader. # --------------------------------------------------------------------------- # TRAINING CRAWLERS — DISALLOWED, AS THE EDGE ALREADY REFUSES THEM (29 September 2026). # Until this date the file said `Allow: /` to GPTBot and ClaudeBot while the edge answered # them 403 on /, /llms.txt and /api/gspc (UA matrix, 29 Sep 10:04Z; CCBot, Bytespider and # Amazonbot also 403). A file that states one policy while the site enforces another is the # declared-versus-observed mismatch this estate measures in others. The file now states what # the edge does. Their user-initiated and search agents (Claude-User, Claude-SearchBot, # ChatGPT-User, OAI-SearchBot) are separate agents, are answered 200, and stay allowed above. # functions/api/robots-agent-paths.test.ts holds the observed matrix and fails on a mismatch. User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / User-agent: ClaudeBot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / User-agent: Bytespider Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / User-agent: Amazonbot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / User-agent: Diffbot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / User-agent: omgili Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / User-agent: Timpibot Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference Disallow: / # --------------------------------------------------------------------------- # Path exclusions are IN THE `*` GROUP ABOVE, not in a group of their own. # # The first draft of this file put them in a second `User-agent: *` block — which is the # precise ambiguity the file exists to remove. The validator caught it. One group per # agent means one, including the wildcard. # # The `.backups/` tree was 240 of 490 URLs in an earlier sitemap and every one of them # served the homepage. It is excluded here as well as removed from the sitemap, because # the two mechanisms fail independently. # --------------------------------------------------------------------------- # --------------------------------------------------------------------------- # Sitemap. # # Points at the www host. The previous file pointed at the apex, and the apex serves the # unprerendered shell on at least some routes, so the sitemap was directing every crawler # to the emptier of the two hosts. Whichever host is chosen as canonical, the sitemap, the # canonical tags and this line must all name the same one. # # BEFORE THIS FILE HELPS ANYTHING: the sitemap must list only URLs that return real # content. As measured on 5 August 2026, roughly 340 of 350 sitemap URLs returned an # 11,279-byte document containing 46 visible characters. A crawler invited in to read # 46 characters is worse than a crawler kept out — it learns that the site is empty. # --------------------------------------------------------------------------- Sitemap: https://councilof.ai/sitemap.xml # The sitemap index names /sitemap.xml above plus one Function-generated sitemap per measured-entity # type (MCP servers, A2A agent cards, x402, tokenised-asset deployments, daily notes). Entities whose # operator opted out, or whose robots.txt refuses CSOAI-census, are not in any of them. Sitemap: https://councilof.ai/sitemaps/index.xml