diff --git a/docs/robots-policy.md b/docs/robots-policy.md new file mode 100644 index 0000000..235120c --- /dev/null +++ b/docs/robots-policy.md @@ -0,0 +1,60 @@ +# Reproducible robots policy + +`site/robots.txt` preserves the exact policy already served publicly by Cloudflare +for `https://oss-singularity.io/robots.txt` on 2026-09-05. This change moves that +existing response into version control. It introduces no new permissions, +restrictions, crawler groups, or training policy. + +## Captured baseline + +- Existing Origin and release `d883ebe174c989d4a278621f86086a9f78bf44bf`: + 72 bytes, SHA-256 + `9f43f4a2c9a9d557c046fd79186287669e22a874344d51e09f6929ce10d480bd`. +- Existing public Edge response, confirmed by two consecutive reads before the + source update: 1,908 bytes, SHA-256 + `c96b21c43640ba554286ca54f225edbb2eadcaae9432a813d9955af42fb76993`. +- Cloudflare prepended 1,836 bytes. The original 72-byte file, including its + `User-agent: *`, `Allow: /`, sitemap URL, and final newline, remains the exact + suffix of the versioned file. + +The preserved response declares +`Content-Signal: search=yes,ai-train=no,use=reference`. It has no explicit +`ai-input` value. Its named `Disallow: /` groups are Amazonbot, +Applebot-Extended, Bytespider, CCBot, ClaudeBot, +CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot, and +meta-externalagent. The Content Signal definitions and Cloudflare marker +comments are retained verbatim as part of the captured response. + +An invitation to participate in the Commons does not change these existing +content-use preferences. Any future policy change needs a separate, explicit +decision and review; this migration does not make that decision. + +## Cutover contract + +1. Review and release this source file through the normal site workflow. Keep + the old release and the current Cloudflare configuration available for + rollback. Do not rewrite an already verified release directory. +2. Verify the new Origin response against the built 1,908-byte artifact before + disabling automatic augmentation. Cloudflare may temporarily prepend its + managed block again while the new Origin and old setting coexist. +3. Disable only the automatic managed-robots augmentation for the OSS + Singularity zone (`is_robots_txt_managed: false`). Preserve all other existing + AI crawler, Bot Management, WAF, and content-policy settings. Compare the + relevant configuration before and after the update. If another setting still + transforms the response, investigate it rather than resetting unrelated + controls. +4. Verify that the public Edge, Origin, and built artifact are byte-identical, + with the captured Edge hash above. Retain the existing exact-byte verifier; + do not strip comments or permit arbitrary provider prefixes to make it pass. +5. On failure, restore the recorded configuration and prior release together. + Subsequent robots policy updates are source-reviewed changes rather than + automatic additions from Cloudflare's managed list. + +Creating these source files does not itself change Cloudflare settings or +perform this cutover. The final public policy must remain the captured policy +throughout the migration; existing crawler enforcement rules remain separate. + +## Primary references + +- [Cloudflare managed robots.txt behavior and Content Signals](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) +- [Cloudflare Bot Management configuration fields](https://developers.cloudflare.com/api/resources/bot_management/methods/update/) diff --git a/site/robots.txt b/site/robots.txt index fd9dc28..856701a 100644 --- a/site/robots.txt +++ b/site/robots.txt @@ -1,3 +1,64 @@ +# As a condition of accessing this website, you agree to abide by the following +# content signals: + +# (a) If a Content-Signal = yes, you may collect content for the corresponding +# use. +# (b) If a Content-Signal = no, you may not collect content for the +# corresponding use. +# (c) If the website operator does not include a Content-Signal for a +# corresponding use, the website operator neither grants nor restricts +# permission via Content-Signal with respect to the corresponding use. + +# The content signals and their meanings are: + +# search: building a search index and providing search results (e.g., returning +# hyperlinks and short excerpts from your website's contents). Search does not +# include providing AI-generated search summaries. +# ai-input: inputting content into one or more AI models (e.g., retrieval +# augmented generation, grounding, or other real-time taking of content for +# generative AI search answers). +# ai-train: training or fine-tuning AI models. +# use: how AI systems may consume the content (immediate, reference, or full). + +# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF +# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT +# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. + +# BEGIN Cloudflare Managed content + +User-agent: * +Content-Signal: search=yes,ai-train=no,use=reference +Allow: / + +User-agent: Amazonbot +Disallow: / + +User-agent: Applebot-Extended +Disallow: / + +User-agent: Bytespider +Disallow: / + +User-agent: CCBot +Disallow: / + +User-agent: ClaudeBot +Disallow: / + +User-agent: CloudflareBrowserRenderingCrawler +Disallow: / + +User-agent: Google-Extended +Disallow: / + +User-agent: GPTBot +Disallow: / + +User-agent: meta-externalagent +Disallow: / + +# END Cloudflare Managed Content + User-agent: * Allow: /