Suhail 2026 source expansion (+ track suhail_pipeline package) - #37
Merged
Conversation
…ackage Records the in-progress package rename src/meshic_pipeline -> src/suhail_pipeline (the package was untracked on disk) and lands this session's Suhail-2026 source audit and integration work on top of it. Source (Stage 1 geometric): - Capture inline market time-series now on the parcels/neighborhoods/subdivisions/ centroids tiles (transaction_price/price_of_meter/transactions_count/ transaction_date at 1w/1m/6m/12m) + zoning_group + Arabic name fixes. - Remap layers whose tile field names changed (bus_lines captured zero attributes; metro_stations/riyadh_bus_stations coords; qi_population_metrics metric fields). - New layers: non_saudi_ownership_zones (unique id upsert); dimensions (tile-scoped delete+append via source_tile, keyless multi-row); building_detection (synthetic bd_id upsert). Two new accumulate write modes: write_tile_scoped + compute_synthetic_pk / SYNTHETIC_PK_CONFIG / TILE_SCOPED_LAYERS. Enrichment (Stage 2): - Promote transaction attributes out of raw_data into queryable columns; fix the building-rules dedup that collapsed all rules per parcel; capture price-metrics neighborhood_id (+ neighborhood stub insertion); split parsers into pure functions. Schema/tests/docs: - Alembic migrations c7f4a9d21b60 (market fields, reshaped layers, new tables) and d8b1e6f42a90 (dimensions + building_detection). Both reversible. - Contract tests over real captured fixtures (tests/fixtures/suhail_live_2026_07/) and new-layer key/write tests. Full unit suite: 93 passed. - docs/SUHAIL_SOURCE_AUDIT_2026-07.md: evidence-led audit + gap analysis. Note: this commit carries the suhail_pipeline package as it currently stands on disk (pre-existing uncommitted rename content plus this session's edits, which were not separable since the package was untracked). The repo's other ~86 pre-existing modified files (dated 2026-03-20) are intentionally left unstaged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The tile server can return an HTML error/maintenance page instead of a vector
tile; decode would otherwise raise a protobuf DecodeError. Add an early guard in
MVTDecoder.decode_bytes that returns {} when the payload starts with '<' (MVT is
binary protobuf and never does), plus tests. Salvaged and cleaned up from the
abandoned fix-mvt-decoder-str-bug branch (dropped its print/debug and hardcoded
province-id hack).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…ject Companion to the package-directory rename in this branch: updates every remaining `meshic`/`Meshic` reference to `suhail`/`Suhail` (pyproject entry points + project name, 14 test imports, README, docs, scripts). Without this the committed branch was internally inconsistent — tracked tests still imported `meshic_pipeline` and pyproject declared the old console scripts, so a clean checkout would fail the suite and install a broken CLI. Also gitignores the `.gstack/` tool dir. Pure rename (1703 meshic->suhail line swaps) plus the one .gitignore line; no behavioural change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ranch) as a demonstrated-need-gated future option Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands this session's evidence-led Suhail source audit + integration work, and records the in-progress
src/meshic_pipeline→src/suhail_pipelinepackage rename (the package was untracked on disk).What changed
Stage 1 (geometric tiles)
transaction_price/price_of_meter/transactions_count/transaction_dateat 1w/1m/6m/12m) +zoning_group+ Arabic-name fixes.bus_lineshad captured zero attributes;metro_stations/riyadh_bus_stationscoords;qi_population_metricsmetric fields).non_saudi_ownership_zones(unique-id upsert);dimensions(tile-scoped delete+append viasource_tile);building_detection(syntheticbd_idupsert). Two new accumulate write modes:write_tile_scoped+compute_synthetic_pk.Stage 2 (enrichment)
raw_datainto queryable columns; fix building-rules dedup that collapsed all rules per parcel; capture price-metricsneighborhood_id(+ neighborhood stub insertion); split parsers into pure functions.Schema / tests / docs
c7f4a9d21b60(market fields, reshaped layers, new tables) andd8b1e6f42a90(dimensions + building_detection); both reversible (round-trip verified).tests/fixtures/suhail_live_2026_07/) + new-layer key/write tests + decoder guard. 96 unit tests pass.docs/SUHAIL_SOURCE_AUDIT_2026-07.md: full audit + gap analysis.Validation
Live E2E against
tiles.suhail.ai/api2.suhail.ai+ local Postgres 18: Stage-1 (7,542 parcels, new columns populated, idempotent, source-vs-persisted cross-check matched the live API), fixed layers, Stage-2 enrichment, and multi-batch accumulation + idempotency fordimensions/building_detection.Note for reviewers
This branch carries the
suhail_pipelinepackage as it stands on disk (pre-existing uncommitted rename content + this session's edits — not separable, since the package was untracked). The repo's other ~86 pre-existing modified files are intentionally not included.🤖 Generated with Claude Code