markdown-to-bots

Serve HTML to humans, Markdown to bots — from a single Hugo static site on AWS.

Verified crawlers (search engines, AI bots, etc.) that send Accept: text/markdown receive raw .md source files. All other visitors get the normal rendered HTML. Social media bots are excluded from markdown serving so Open Graph tags remain intact for link previews.

How it works

Request → CloudFront → WAF (Bot Control) → CloudFront Function → S3

1. WAF Bot Control (Targeted) inspects each request and labels verified bots by category (search_engine, ai, advertising, content_fetcher, seo).
2. A WAF rule matches verified + harmless-category bots and inserts the header x-amzn-waf-verified-bot: true into the request.
3. The CloudFront Function (url-rewrite.js) reads this header plus the client's Accept header:
   • Verified bot + Accept: text/markdown → rewrites URI to /md-content/<path>/index.md
   • Everything else → standard index.html rewrite for clean URLs
4. Direct access to /md-content/ by non-bots returns 403 Forbidden.
5. The CloudFront cache key includes both x-amzn-waf-verified-bot and x-wants-markdown headers, so HTML and Markdown responses are cached separately.

Architecture

Component	Details
S3	Private bucket; access only via CloudFront OAC
CloudFront	CDN with custom cache policy; CloudFront Function on viewer-request
WAF	Bot Control (Targeted v5.0), IP reputation list, anonymous IP list, anti-DDoS, core rule set, known bad inputs, spoofed-header blocking
ACM	TLS certificate with automatic DNS validation via Route 53
Route 53	A + AAAA alias records pointing to the CloudFront distribution

All resources are provisioned in us-east-1 (required by CloudFront, WAF, and ACM).

Repository structure

infra/
  main.tf                  # Terraform provider + version config
  variables.tf             # domain_name, route53_zone_name
  s3.tf                    # S3 bucket + OAC policy
  cloudfront.tf            # CloudFront distribution, cache policy, CF Function
  acm.tf                   # ACM certificate + Route 53 DNS validation + alias records
  waf.tf                   # WAFv2 Web ACL with all rules + CloudWatch logging
  outputs.tf               # cloudfront_distribution_id, s3_bucket_name, site_url
  terraform.tfvars.example # Example variable values
  cf-function/
    url-rewrite.js         # CloudFront Function: bot detection + URI rewrite logic
scripts/
  deploy.sh                # Build Hugo → copy markdown → sync S3 → invalidate CloudFront

The deploy script expects a hugo/ directory at the project root containing the Hugo site. Markdown source files are copied to hugo/public/md-content/ before syncing to S3.

Prerequisites

• Terraform >= 1.0
• AWS CLI configured with appropriate credentials
• Hugo installed
• A Route 53 hosted zone for your domain

Setup

1. Configure variables

cp infra/terraform.tfvars.example infra/terraform.tfvars

Edit infra/terraform.tfvars:

domain_name      = "example.yourdomain.com"
route53_zone_name = "yourdomain.com"

2. Provision infrastructure

cd infra
terraform init
terraform apply

ACM certificate validation and CloudFront distribution creation can take several minutes.

3. Deploy the site

bash scripts/deploy.sh

This will:

1. Build the Hugo site (hugo --minify)
2. Copy markdown source files to hugo/public/md-content/
3. Read S3 bucket name and CloudFront distribution ID from Terraform outputs
4. Sync the built site to S3
5. Invalidate the CloudFront cache

WAF rules (in priority order)

Priority	Rule	Action
0	Block spoofed x-amzn-waf-* headers	Block
1	Allowed IP list	Allow
2	Denied IP list	Block
3	AWS IP Reputation List	Block (DDoS list)
4	Anonymous IP List	Managed default
5	Anti-DDoS	Managed default
6	Core Rule Set	Managed default
7	Known Bad Inputs	Managed default
8	Bot Control (Targeted)	CategoryAI → Count (to allow verified AI bots through)
9	Label verified harmless bots	Count + insert verified-bot: true header

Static assets (.css, .js, images, fonts) are excluded from Bot Control inspection to reduce costs.

Notes

• Social media bots (e.g. Twitter/X, Facebook) are intentionally excluded from markdown serving. They need the rendered HTML page so Open Graph meta tags work for link previews.
• The WAF header inserted by rule #9 is named verified-bot; AWS WAF automatically prepends x-amzn-waf- making it x-amzn-waf-verified-bot in the CloudFront Function.
• Spoofed x-amzn-waf-* headers from clients are blocked at priority 0 before any bot-labeling rules run.