Review: Compass environment, source-backed data and task verification

.assets-revision

@@ -1,8 +1 @@
-# Hugging Face dataset holding webharbor's heavy static assets
-# (instance_seed/*.db, static/images/, static/external_cache/).
-#
-# fetch_assets.sh uses the `hf download` CLI. The pin below
-# is a git revision (branch name like `main`, a tag, or a specific commit
-# sha). Override at runtime with the ASSETS_REVISION env var.
-repo: ChilleD/WebHarbor
-revision: 8d8e4069588ef55594622fee1ab9fa51c2011d07
+d86ec0bfcbb98f92efbe6b6b4440f9dcc6cceb69

AGENTS.md

@@ -4,7 +4,7 @@ A coding agent (Claude Code, Cursor, Aider, Codex, ...) is reading this. Read on
 
 ## What it is
 
-19 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
+20 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
 
 Two repos:
 - **code** (this one) — Flask apps, control plane, scripts.
@@ -48,17 +48,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
 # fresh clone
 ./scripts/fetch_assets.sh                     # pulls assets from HF
 ./scripts/build.sh                            # docker build -t webharbor:dev .
-docker run -d -p 8101:8101 -p 40000-40018:40000-40018 webharbor:dev
+docker run -d -p 8101:8101 -p 40000-40019:40000-40019 webharbor:dev
 ```
 
 Or use the published image directly:
 
 ```bash
-docker run -d -p 8101:8101 -p 40000-40018:40000-40018 \
+docker run -d -p 8101:8101 -p 40000-40019:40000-40019 \
   battalion7244/webharbor:latest
 ```
 
-Sites are on `40000`-`40018` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
+Sites are on `40000`-`40019` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
 
 | Method | Path                | Purpose                                   |
 |--------|---------------------|-------------------------------------------|
@@ -136,13 +136,13 @@ python3 -m py_compile sites/<site>/app.py
 
 # 3. run on alt ports (don't collide with anything you already have running)
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41018:40000-40018 webharbor:dev
+  -p 8201:8101 -p 41000-41019:40000-40019 webharbor:dev
 
 # 4. control plane healthy, all sites alive
 curl -s http://localhost:8201/health | python3 -m json.tool | head
 
 # 5. every site renders 200
-for p in $(seq 41000 41018); do
+for p in $(seq 41000 41019); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

CLAUDE.md

@@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl
 
 ## Existing containers
 
-If a container is already running on `:8101` / `:40000-40018`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41018`).
+If a container is already running on `:8101` / `:40000-40019`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41019`).

CONTRIBUTING.md

@@ -24,7 +24,7 @@ git clone https://github.com/<you>/webharbor && cd webharbor
 ./scripts/fetch_assets.sh                       # pull current assets
 ./scripts/new_site.py mywebsite                 # OR edit an existing site
 ./scripts/build.sh && docker run -d --rm \
-  -p 8101:8101 -p 40000-40018:40000-40018 webharbor:dev
+  -p 8101:8101 -p 40000-40019:40000-40019 webharbor:dev
 # iterate locally...
 
 ./scripts/extract_assets.sh ../webharbor-static-pr/   # split assets out

Dockerfile

@@ -1,5 +1,5 @@
 # WebHarbor — slim, self-contained image.
-# 19 Flask mirror sites + control plane on :8101.
+# 20 Flask mirror sites + control plane on :8101.
 
 FROM python:3.12-slim-bookworm
 
@@ -35,11 +35,14 @@ RUN cd /opt/WebSyn/ikea && PYTHONHASHSEED=0 python seed_data.py && rm -rf instan
 RUN cd /opt/WebSyn/phys_org && PYTHONHASHSEED=0 python migrate_seed.py && rm -rf instance
 RUN cd /opt/WebSyn/target && PYTHONHASHSEED=0 python migrate_seed.py && rm -rf instance
 
+# Rebuild Compass's source-backed catalog and benchmark state from tracked data.
+RUN cd /opt/WebSyn/compass && python migrate_seed.py && rm -rf instance
+
 COPY websyn_start.sh    /opt/websyn_start.sh
 COPY control_server.py  /opt/control_server.py
 COPY site_runner.py     /opt/site_runner.py
 RUN chmod +x /opt/websyn_start.sh
 
-EXPOSE 8101 40000-40018
+EXPOSE 8101 40000-40019
 
 CMD ["/opt/websyn_start.sh"]

README.md

@@ -36,17 +36,17 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
 - **Deep features unlocked** — carts, checkouts, accounts, all fully testable
 - **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
 - **RL-ready** — sub-second database resets between rollouts
-- **Community-driven** — 19 sites today, scaling to 100+ together
+- **Community-driven** — 20 sites today, scaling to 100+ together
 
 ## 🚀 Quickstart
 
 One command to run all web environments:
 
 ```bash
-docker run -p 8101:8101 -p 40000-40018:40000-40018 battalion7244/webharbor:latest
+docker run -p 8101:8101 -p 40000-40019:40000-40019 battalion7244/webharbor:latest
 ```
 
-Then point your agent at `http://localhost:40000` through `http://localhost:40018` to explore 19 local mirrors of webvoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, and Target`.
+Then point your agent at `http://localhost:40000` through `http://localhost:40019` to explore 20 local mirrors of webvoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, and Compass`.
 
 For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:
 
@@ -65,7 +65,7 @@ git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor
 
 ## 🤝 Contribute
 
-We have built 18 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.
+We have built 19 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.
 
 There are two ways to join the author list:
 

agent_demo/README.md

@@ -19,7 +19,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1   # or your Azure / vLLM endpoi
 
 ## Run a task
 
-WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40018:40000-40018 battalion7244/webharbor:latest`).
+WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40019:40000-40019 battalion7244/webharbor:latest`).
 
 Run a single task from a site's `tasks.jsonl`:
 

control_server.py

@@ -26,7 +26,7 @@
     'allrecipes', 'amazon', 'apple', 'arxiv', 'bbc_news', 'booking',
     'github', 'google_flights', 'google_map', 'google_search',
     'huggingface', 'wolfram_alpha', 'cambridge_dictionary',
-    'coursera', 'espn', 'merriam_webster', 'ikea', 'phys_org', 'target',
+    'coursera', 'espn', 'merriam_webster', 'ikea', 'phys_org', 'target', 'compass',
 ]
 BASE_PORT = 40000
 WEBSYN_DIR = '/opt/WebSyn'

sites/compass/README.md

@@ -0,0 +1,53 @@
+> **Current UI follow-up:** [Sell and all 24 regional guide landings](docs/sell-guides-review.md). Human experience and disclosed scope are accepted; [maintainer handoff and remaining work](docs/maintainer-handoff.md) record the Ready request and outstanding refreshed task17 verdict. Earlier evidence retains its recorded versions.
+
+# Compass
+
+An offline Compass mirror contributed in [WebHarbor #25](https://github.com/aiming-lab/WebHarbor/pull/25), with browsing, accounts, saved homes, saved searches, collections, tour requests, and inquiries. Compass uses port **40019** in the current 20-site registry.
+
+## Data and assets
+
+- `listings_clean.json` preserves the contributor's 524 source records and listing IDs. The environment includes the 497 records with an available local gallery.
+- `source_data.json` contains 312 public Compass detail snapshots checked on September 6, 2026. Of these, 295 have local galleries and appear in the environment. Each record includes its original listing ID, source URL, retrieval time, and HTML SHA-256.
+- Detail facts are accepted only when the public page's listing ID matches the original transaction. A URL that now resolves to a different sale or rental is not used to enrich the old record. Unmatched records retain only the contributor's basic snapshot; unknown property details and agent information stay absent.
+- Property type, year, MLS number, amenities, agent contact information, and availability are never generated. A missing amenity is unknown, rather than false. Historical rentals retain monthly price formatting. Missing prices are displayed as “Price upon request.”
+- `gallery_sources.json` records the additional original-source gallery images. `visual_sources.json` records the official homepage, Sell, Concierge and directory media/font assets. Local files are distributed through the pinned Hugging Face asset archive, not downloaded by the running site.
+- Only benchmark accounts, their preferences, saved homes, collections, searches, and tour requests are synthetic. Forms save state locally; they do not contact real agents.
+
+The contributor's generation of property years, MLS numbers, amenities, and agent sales statistics has been removed. Published listing details take precedence over assessor records when those sources disagree; they are not silently combined.
+
+`migrate_seed.py` rebuilds `instance_seed/compass.db` from the tracked catalog and available local images. It writes tables and indexes in a stable order for byte-reproducible seeds within the same SQLite runtime and never modifies the live `instance/compass.db`. Both asset fetch and the Docker build apply this migration. Different SQLite versions can encode equivalent rows differently; reset comparisons use the seed rebuilt inside the same runtime.
+
+## Local use
+
+From the repository root, install the site's requirements and fetch its pinned assets:
+
+```bash
+python -m pip install -r sites/compass/requirements.txt huggingface_hub
+./scripts/fetch_assets.sh compass
+mkdir -p sites/compass/instance
+cp sites/compass/instance_seed/compass.db sites/compass/instance/compass.db
+PORT=40019 python sites/compass/app.py
+```
+
+The default database can be overridden with `COMPASS_DATABASE_PATH`. `COMPASS_SKIP_SEED=1` is reserved for isolated tests. The full environment should be started with the repository's Docker and control-plane workflow so that reset restores the complete seed.
+
+Benchmark logins use password `webharbor123`: `alice.j@test.com`, `bob.smith@test.com`, `carol.lee@test.com`, and `david.kim@test.com`. Tour dates remain valid independently of wall-clock time because benchmark requests use fixed dates.
+
+```bash
+python -m pip install pytest
+python -m pytest sites/compass/tests -q
+```
+
+Tests cover invalid input without partial writes, object ownership, CSRF, local redirects, saved-search and collection round trips, rental formatting, price-per-area sorting, source transaction identity, and reproducible seed layout. They are regression tests, not recorded agent trajectories.
+
+## Interface scope
+
+The homepage hero, official typography, navigation, listing cards, responsive layout, photo wall, gallery, and filter controls follow the public site. Personalized recommendations are replaced with a stable catalog selection. Search cards do not add property years, MLS numbers, or agent identities; detail questions still require obtaining the relevant information.
+
+Sell at `/sell/` reproduces the source marketing sequence, both carousels and two local inquiry forms. All 24 directory entries lead to local regional guide landings, with 401 original community cards; individual articles and nearby searches remain explicit external links. See the [source/before/after comparison and scope](docs/sell-guides-review.md).
+
+The mirror provides a list view. Live maps, street view, mortgage preapproval, property-history feeds, and external messaging are not implemented. These are functional scope limits, not measures to prevent answer leakage. Unknown facts are identified rather than filled with plausible values. The displayed catalog is a snapshot and can include sold or rented homes.
+
+In this fixed snapshot, the **For sale** filter includes Pending / Contract Signed listings in the sale category; their published status remains in property details. It is not an active-only filter. A named search area includes records whose address city or catalog market matches that area, so New York can include Brooklyn and Manhattan addresses.
+
+The revised task set preserves the original IDs and omits `Compass--8` and `Compass--9`, which depended on generated agent sales volumes or unsupported open-house schedules. Remaining tasks use source-backed detail facts or explicit local account changes.

sites/compass/_health.py

@@ -0,0 +1,36 @@
+"""compass mirror health check."""
+from healthcheck import random_user
+
+
+def run(p):
+    p.assert_get('home', '/', must_contain='Compass')
+    p.assert_get('search New York', '/search?q=New York', must_contain='New York')
+    p.assert_get('city for-sale', '/homes-for-sale/new-york-ny/',
+                 must_contain='Homes')
+    p.assert_get('agents directory', '/agents', must_contain='Agent')
+    p.assert_get('open houses', '/open-houses', must_contain='Open')
+
+    user = random_user()
+    html = p.assert_get('register page', '/register')
+    token = p.csrf(html)
+    if not token:
+        p.check('register csrf', False, 'no csrf'); return
+    p.assert_post('register submit', '/register', {
+        'csrf_token': token,
+        'name':     f"{user['first_name']} {user['last_name']}",
+        'email':    user['email'],
+        'password': user['password'],
+        'confirm':  user['password'],
+    }, accept_status=(200, 302, 303))
+
+    p.get('/logout')
+    html = p.assert_get('login page', '/login')
+    token = p.csrf(html)
+    if not token:
+        p.check('login csrf', False, 'no csrf'); return
+    p.assert_post('login submit', '/login', {
+        'csrf_token': token,
+        'email':    user['email'],
+        'password': user['password'],
+    }, accept_status=(200, 302, 303))
+    p.assert_get('account', '/account', accept_status=(200, 302))