Warn instead of blocking when a spider name differs from its domain#7
Open
MirjamOdile wants to merge 1 commit into
Open
Warn instead of blocking when a spider name differs from its domain#7MirjamOdile wants to merge 1 commit into
MirjamOdile wants to merge 1 commit into
Conversation
The importer hard-rejected any spider whose name didn't equal the domain-derived folder name whenever source_url was set. But a domain can legitimately host more than one spider (a second template, a sub-domain grouped under the main site), and crawls + analysis both key off the spider NAME consistently — nothing actually scatters. The check forced legit sub-spiders to drop source_url just to get imported. Keep the check as a warning that states where the data will live. See docs/requests/05-spider-name-domain-warn.md (ships with the quality tool PR) for the full write-up.
iRanadheer
force-pushed
the
pr/02-spider-name-warn
branch
from
8447b59 to
c9cae66
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
spiders importhard-rejects any config whose spider name doesn't equal thedomain-derived name whenever
source_urlis set. But a domain can legitimatelyhost more than one spider — a second template on the same site (e.g.
noaa_gov+noaa_gov_gc), or a sub-domain grouped under the main site(
ncei.noaa.govasnoaa_gov_ncei). The check forced those legit sub-spidersto drop
source_urljust to get imported, losing the provenance field.Change
The mismatch is now a warning instead of a rejection. It states exactly where
the data will live: crawls under
data/<project>/<spider_name>/, while theinspector's analysis folder stays domain-derived. Everything downstream
(crawl output paths, DeltaFetch cache, checkpoint, exports, DB attribution)
keys off the spider NAME consistently, so nothing scatters; the honest
mismatch-shaped-folder split (crawls vs analysis) is named in the warning.
Verification
Full unit suite green. Exercised live: importing a secondary spider for an
already-covered domain now succeeds with the warning, with
source_urlkept.