feat(walmart_careers): add Walmart Careers mirror (site 20, port 40019)

.assets-revision

@@ -5,4 +5,4 @@
 # is a git revision (branch name like `main`, a tag, or a specific commit
 # sha). Override at runtime with the ASSETS_REVISION env var.
 repo: ChilleD/WebHarbor
-revision: 8d8e4069588ef55594622fee1ab9fa51c2011d07
+revision: refs/pr/54

.dockerignore

@@ -8,6 +8,9 @@ sites/*/instance/
 # Don't ship — runtime doesn't need scrape intermediate (data is in instance_seed/*.db).
 sites/*/scraped_data/
 
+# Don't ship — per-site local dev helpers (asset harvest, smoke tests, leak audits).
+sites/*/scripts_dev/
+
 # Don't ship — bytecode / venvs.
 sites/*/__pycache__/
 **/__pycache__/

.gitignore

@@ -11,6 +11,7 @@ sites/*/static/external_cache/
 # =============================================================
 # scrape pipeline intermediate; runtime data lives in instance_seed/*.db
 sites/*/scraped_data/
+/scraped_data/
 # rebuilt at every container boot from instance_seed/
 sites/*/instance/
 sites/*/venv/

AGENTS.md

@@ -4,7 +4,7 @@ A coding agent (Claude Code, Cursor, Aider, Codex, ...) is reading this. Read on
 
 ## What it is
 
-19 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
+20 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
 
 Two repos:
 - **code** (this one) — Flask apps, control plane, scripts.
@@ -48,17 +48,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
 # fresh clone
 ./scripts/fetch_assets.sh                     # pulls assets from HF
 ./scripts/build.sh                            # docker build -t webharbor:dev .
-docker run -d -p 8101:8101 -p 40000-40018:40000-40018 webharbor:dev
+docker run -d -p 8101:8101 -p 40000-40019:40000-40019 webharbor:dev
 ```
 
 Or use the published image directly:
 
 ```bash
-docker run -d -p 8101:8101 -p 40000-40018:40000-40018 \
+docker run -d -p 8101:8101 -p 40000-40019:40000-40019 \
   battalion7244/webharbor:latest
 ```
 
-Sites are on `40000`-`40018` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
+Sites are on `40000`-`40019` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
 
 | Method | Path                | Purpose                                   |
 |--------|---------------------|-------------------------------------------|
@@ -136,13 +136,13 @@ python3 -m py_compile sites/<site>/app.py
 
 # 3. run on alt ports (don't collide with anything you already have running)
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41018:40000-40018 webharbor:dev
+  -p 8201:8101 -p 41000-41019:40000-40019 webharbor:dev
 
 # 4. control plane healthy, all sites alive
 curl -s http://localhost:8201/health | python3 -m json.tool | head
 
 # 5. every site renders 200
-for p in $(seq 41000 41018); do
+for p in $(seq 41000 41019); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

CLAUDE.md

@@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl
 
 ## Existing containers
 
-If a container is already running on `:8101` / `:40000-40018`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41018`).
+If a container is already running on `:8101` / `:40000-40019`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41019`).

CONTRIBUTING.md

@@ -24,7 +24,7 @@ git clone https://github.com/<you>/webharbor && cd webharbor
 ./scripts/fetch_assets.sh                       # pull current assets
 ./scripts/new_site.py mywebsite                 # OR edit an existing site
 ./scripts/build.sh && docker run -d --rm \
-  -p 8101:8101 -p 40000-40018:40000-40018 webharbor:dev
+  -p 8101:8101 -p 40000-40019:40000-40019 webharbor:dev
 # iterate locally...
 
 ./scripts/extract_assets.sh ../webharbor-static-pr/   # split assets out

Dockerfile

@@ -1,5 +1,5 @@
 # WebHarbor — slim, self-contained image.
-# 19 Flask mirror sites + control plane on :8101.
+# 20 Flask mirror sites + control plane on :8101.
 
 FROM python:3.12-slim-bookworm
 
@@ -40,6 +40,6 @@ COPY control_server.py  /opt/control_server.py
 COPY site_runner.py     /opt/site_runner.py
 RUN chmod +x /opt/websyn_start.sh
 
-EXPOSE 8101 40000-40018
+EXPOSE 8101 40000-40019
 
 CMD ["/opt/websyn_start.sh"]

README.md

@@ -36,17 +36,17 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
 - **Deep features unlocked** — carts, checkouts, accounts, all fully testable
 - **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
 - **RL-ready** — sub-second database resets between rollouts
-- **Community-driven** — 19 sites today, scaling to 100+ together
+- **Community-driven** — 20 sites today, scaling to 100+ together
 
 ## 🚀 Quickstart
 
 One command to run all web environments:
 
 ```bash
-docker run -p 8101:8101 -p 40000-40018:40000-40018 battalion7244/webharbor:latest
+docker run -p 8101:8101 -p 40000-40019:40000-40019 battalion7244/webharbor:latest
 ```
 
-Then point your agent at `http://localhost:40000` through `http://localhost:40018` to explore 19 local mirrors of webvoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, and Target`.
+Then point your agent at `http://localhost:40000` through `http://localhost:40019` to explore 20 local mirrors of webvoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, and Walmart Careers`.
 
 For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:
 
@@ -65,7 +65,7 @@ git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor
 
 ## 🤝 Contribute
 
-We have built 18 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.
+We have built 20 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.
 
 There are two ways to join the author list:
 

agent_demo/README.md

@@ -19,7 +19,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1   # or your Azure / vLLM endpoi
 
 ## Run a task
 
-WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40018:40000-40018 battalion7244/webharbor:latest`).
+WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40019:40000-40019 battalion7244/webharbor:latest`).
 
 Run a single task from a site's `tasks.jsonl`:
 

control_server.py

@@ -27,6 +27,7 @@
     'github', 'google_flights', 'google_map', 'google_search',
     'huggingface', 'wolfram_alpha', 'cambridge_dictionary',
     'coursera', 'espn', 'merriam_webster', 'ikea', 'phys_org', 'target',
+    'walmart_careers',
 ]
 BASE_PORT = 40000
 WEBSYN_DIR = '/opt/WebSyn'

sites/walmart_careers/.gitignore

@@ -0,0 +1,2 @@
+# Local build/inspection helpers: never shipped with the site.
+scripts_dev/

sites/walmart_careers/CLAUDE.md

@@ -0,0 +1,80 @@
+# walmart_careers — site notes
+
+Mirror of https://careers.walmart.com. Port **40019** (index 19 in `websyn_start.sh`, the 20th and last site);
+alt-port test container maps it to **41019**.
+
+## Layout
+
+| file | role |
+|---|---|
+| `app.py` | models, routes, scored search, deterministic SVG maps, bootstrap |
+| `catalog_source.py` | the source catalog: areas, categories, 51 stores, 37 hourly + 29 salaried title families with explicit placements, hub copy, trending job ids |
+| `seed_data.py` | turns the catalog into SQLite; `build_seed_database()` is the freezer |
+| `_content.py` | static chrome strings only: headings, boilerplate prose, design constants, US/PR map outlines |
+| `templates/` | 20 Jinja2 templates (`base.html`, the stripped `base_auth.html` for sign-in/register) + the `_job_card.html`, `_benefits.html` and `_life.html` macros shared by the detail, career-area and About pages |
+| `static/` | `css/`, `js/`, `icons/`, `fonts/` in git; `images/` HF-managed |
+| `scripts_dev/` | local-only helpers and build-time invariants; gitignored and dockerignored |
+
+Everything a handler renders about a job, a store or a hub comes from SQLAlchemy.
+`_content.py` holds no per-record content: hub name/blurb/image live on `Store`,
+and the trending flag lives on `Job.is_trending`.
+
+## Rebuilding the seed DB
+
+```bash
+cd sites/walmart_careers
+PYTHONHASHSEED=0 python seed_data.py     # writes instance_seed/walmart_careers.db
+```
+
+Run it twice and compare md5s — the build is byte-reproducible.
+
+`build_seed_database()` also runs the build-time invariant checks, which fail the
+build if a catalog edit breaks a volume invariant (jobs per category/store/state/
+shift) or a benchmark task's near-miss set. Those checks live in
+`scripts_dev/assert_distractors.py`, which the freezer loads by path *only if the
+file exists* — it is git-ignored and docker-ignored, so it never reaches the shipped
+tree, and a checkout without it builds the identical database and prints a note that
+the checks were skipped. Nothing in `seed_data.py` or `app.py` encodes what a task
+is looking for. The checks never run at import, bootstrap or `/reset` time.
+
+## Determinism rules that must hold
+
+- one RNG: `random.Random(20260905)` in `seed_data.py`, nothing else
+- `MIRROR_REFERENCE_DATE` instead of `date.today()`; no `utcnow()`/`now()` anywhere on the
+  import or bootstrap path (runtime writes may use `now()` — `/reset` wipes them)
+- the four werkzeug password hashes are hard-coded (werkzeug salts randomly)
+- `seed_database()` and `seed_benchmark_users()` are each gated as a whole
+
+## Local dev helpers (`scripts_dev/`, never shipped)
+
+```bash
+python scripts_dev/serve.py 5017            # run with Jinja auto-reload on
+python scripts_dev/walkthrough.py <base_url> [<control-plane reset url>]
+python scripts_dev/leak_audit.py  <base_url>     # writes leak_audit.md
+python scripts_dev/robustness.py  <base_url>
+python scripts_dev/shots.py       <base_url> <out_dir>   # 1440px screenshots
+```
+
+`assert_distractors.py` is not run directly — the freezer imports it by path.
+
+`assert_distractors.py`, `walkthrough.py`, `leak_audit.py`, `robustness.py` and
+`VERIFICATION.md` hold the ground-truth answers, which is exactly why `scripts_dev/`
+is in both `.gitignore` and `.dockerignore`.
+
+## Assets
+
+Brand chrome (`static/icons/`, `static/fonts/`) is committed; photography
+(`static/images/`) is HF-managed. Everything was pulled from the live site:
+`cms.careers.walmart.com/content/dam/careers/...`, `careers.walmart.com/assets/svgs/...`,
+and the `EverydaySansUI` / `LivingDesign` font files from `i5.walmartimages.com`.
+`scripts_dev/harvest_assets.py` records the exact URL → filename mapping and re-downloads
+them; images are then downscaled to 1600px wide. Upstream carries no Dallas hub photo, so
+`loc-dallas.jpg` is the Global Tech office interior from the same CMS (see `harvest_assets.py`).
+
+## Things that are deliberately not mirrored
+
+Google Maps (replaced by a deterministic server-rendered SVG cluster map on `/results`
+and an SVG pin card on the detail page), the LLM search assistant (replaced by ordinary
+query params with token-overlap scoring), Workday/OIDC login (local email + password),
+the OTP apply flow (two-step local form), and the Future roles / Content tabs (explicit
+empty-state panels).