Background
The static robots.txt is a production artefact (Allow: /) that gets served on all builds — staging and forks alike. This is fine in practice today:
- Staging (
wwwstage.ibm.com) — Akamai injects x-robots-tag: noindex, nofollow on every response, which takes precedence over robots.txt for all major crawlers
- Forks (GitHub Pages) — no Akamai protection, so
Allow: / technically permits crawling of fork preview URLs, though in practice they won't be discovered
The subtle risk is that the noindex protection on staging is entirely Akamai-side and invisible in the repo. If the Akamai rule ever changes or a new non-IBM hosting environment is added (e.g. a partner staging domain), the Docusaurus build itself provides no protection.
Suggested fix
Inject a <meta name="robots" content="noindex"> tag for non-production builds via docusaurus.config.ts:
customFields: { noIndex: process.env.DOCS_SITE_URL !== 'https://www.ibm.com' },And consume it in a swizzled <Head> or via the noIndex Docusaurus config option:
noIndex: process.env.DOCS_SITE_URL !== 'https://www.ibm.com',
This makes the noindex intent explicit in the codebase and independent of Akamai, with zero impact on production.
Related
Background
The static
robots.txtis a production artefact (Allow: /) that gets served on all builds — staging and forks alike. This is fine in practice today:wwwstage.ibm.com) — Akamai injectsx-robots-tag: noindex, nofollowon every response, which takes precedence overrobots.txtfor all major crawlersAllow: /technically permits crawling of fork preview URLs, though in practice they won't be discoveredThe subtle risk is that the noindex protection on staging is entirely Akamai-side and invisible in the repo. If the Akamai rule ever changes or a new non-IBM hosting environment is added (e.g. a partner staging domain), the Docusaurus build itself provides no protection.
Suggested fix
Inject a
<meta name="robots" content="noindex">tag for non-production builds viadocusaurus.config.ts:And consume it in a swizzled
<Head>or via thenoIndexDocusaurus config option:This makes the noindex intent explicit in the codebase and independent of Akamai, with zero impact on production.
Related
🤖 Generated with Claude Code