🧠 Context
The scraper already fetches and extracts calendar.carleton.ca pages — it just doesn't work well on the catalog templates. _extract_from_html in src/ingestion/scrapers/html_scraper.py scopes content with one selector, soup.select_one("main, [role=main], #main-content"), which covers the Drupal carleton.ca pages (<main>), the CCSS site (falls back to <body>), and the calendar's CourseLeaf template (<div id="main-content">).
The problem is specific to the calendar's program and course-listing pages. There, #main-content wraps the entire left-hand catalog navigation ("Programs / Courses / Regulations / …") and hundreds of course cross-links, and none of that nav's class names are in _BOILERPLATE_SELECTORS.
Scope is the calendar catalog pages only. Don't change how the extractor handles the existing carleton.ca / CCSS templates.
Files this ticket owns:
src/ingestion/scrapers/html_scraper.pytests/ingestion/test_scraper.py- two new fixtures under
tests/ingestion/fixtures/
🛠 Implementation Plan
1. Investigate the real HTML first
Before changing anything, look at the actual HTML of four representative calendar pages — they span the two behaviours (noisy catalog vs. clean prose), and comparing them is what tells you where the real content lives:
- Academic year:
https://calendar.carleton.ca/academicyear/
- COMP courses:
https://calendar.carleton.ca/undergrad/courses/COMP/
- CS program:
https://calendar.carleton.ca/undergrad/undergradprograms/computerscience/
- A regulations page (already extracts fine):
https://calendar.carleton.ca/undergrad/regulations/academicregulationsoftheuniversity/grading/
The content of these pages is in the static HTML (this is not a JS-rendered site), so curl / "save page as" is enough to inspect and to capture fixtures. Work out what distinguishes the real content (course descriptions, program requirements) from the catalog side-nav and the dense course cross-link lists — e.g. whether there's an inner container tighter than #main-content, and what class/element the nav menu uses. Let what you find drive the fix; don't guess selectors up front.
It also helps to see the current output first: run the scraper's scrape() against a couple of the existing (non-calendar) URLs from data/webpages/list.json and against one of the calendar URLs above, and compare. That shows you what good extraction looks like on the templates that already work, and makes the calendar noise concrete before you start changing anything.
2. Update the extractor — gated to calendar hosts only
Keep all calendar-specific handling behind a hostname check so the other templates run exactly the code they run today. The extractor needs the page's URL to do that, so give _extract_from_html a page_url: str = "" parameter and have scrape() pass the real url. The default keeps existing callers — and the current tests — working unchanged.
Branch on the host (e.g. if "calendar.carleton.ca" in urlsplit(page_url).netloc:) and apply the fix inside that branch only. Based on the investigation, pick whichever fits the real markup (any combination):
- scope tighter than
#main-content to the inner content container the calendar template uses;
- add calendar-specific selectors for the catalog nav/menu;
- strip the dense course-link list blocks if they turn out to be the main source of noise.
- any other improvements you can think of
Because everything is gated on the host, the CCSS and carleton.ca paths stay byte-for-byte identical to today — the shared single-selector scope and the <body> fallback are untouched. Still run the full scraper test to confirm no regression, since the branch lives in shared code.
3. Add fixtures and tests
Capture two fixtures — the CS program page and the COMP courses page (the two noisy cases) — and save them under tests/ingestion/fixtures/ following the existing naming (e.g. calendarProgram.html, calendarCourses.html). The academic-year and regulations pages are for investigation/comparison; they don't need fixtures.
Because the fix is gated on the host, the new test entries must pass the calendar URL into _extract_from_html(html, url) so the branch actually fires — add a url field to those pages entries and pass it in the loop (the existing non-calendar entries have no url and default to "", so they call the extractor exactly as before).
Extend the pages list in tests/ingestion/test_scraper.py with an entry per new fixture, asserting both sides:
- Real content is present — a handful of strings that only appear in the page's actual content (course codes/titles, program requirement text).
- Catalog-nav noise is absent — a string that only appears in the side-nav, not in this page's own content. A good discriminator: the catalog nav lists every program/subject in the university, so a program or subject name unrelated to this page (identified during the investigation) appearing in the extracted text means the nav leaked. Pick that during step 1.
The current test uses a single shared junk list checked against all pages. If a negative assertion only makes sense for the calendar pages, add a per-page exclude list rather than putting a calendar-specific string in the global junk (which would then be wrongly required absent from every other fixture too).
📝 Notes
- Don't regress existing fixtures — run the full scraper test, not just the new cases. The other templates share the same code path.
- Static HTML only — no new dependency, no Playwright. If any target page turns out to hide content behind JS, first flag it to Jacc rather than adding a browser engine.
- bs4 text-extraction gotchas already handled in this file (block-aware newlines,
get_text(" ", strip=...)) still apply — follow the existing patterns.
✅ Acceptance Criteria
_extract_from_html produces clean content for calendar catalog pages: the CS program and COMP courses pages no longer include the catalog side-nav / menu, and the extracted text is dominated by real course/program content rather than nav links.- Calendar-specific handling is gated on the page host (via a new
page_url: str = "" parameter on _extract_from_html, passed through by scrape()), so the CCSS and carleton.ca output is unchanged and the existing fixtures pass with no edits to their assertions.
- Two new fixtures (CS program, COMP courses) are added under
tests/ingestion/fixtures/, with tests that pass the calendar URL in and assert real content is present and catalog-nav noise is absent.
- No new dependency.
make test and make lint pass.
🧠 Context
The scraper already fetches and extracts
calendar.carleton.capages — it just doesn't work well on the catalog templates._extract_from_htmlinsrc/ingestion/scrapers/html_scraper.pyscopes content with one selector,soup.select_one("main, [role=main], #main-content"), which covers the Drupalcarleton.capages (<main>), the CCSS site (falls back to<body>), and the calendar's CourseLeaf template (<div id="main-content">).The problem is specific to the calendar's program and course-listing pages. There,
#main-contentwraps the entire left-hand catalog navigation ("Programs / Courses / Regulations / …") and hundreds of course cross-links, and none of that nav's class names are in_BOILERPLATE_SELECTORS.Scope is the calendar catalog pages only. Don't change how the extractor handles the existing
carleton.ca/ CCSS templates.Files this ticket owns:
src/ingestion/scrapers/html_scraper.pytests/ingestion/test_scraper.pytests/ingestion/fixtures/🛠 Implementation Plan
1. Investigate the real HTML first
Before changing anything, look at the actual HTML of four representative calendar pages — they span the two behaviours (noisy catalog vs. clean prose), and comparing them is what tells you where the real content lives:
https://calendar.carleton.ca/academicyear/https://calendar.carleton.ca/undergrad/courses/COMP/https://calendar.carleton.ca/undergrad/undergradprograms/computerscience/https://calendar.carleton.ca/undergrad/regulations/academicregulationsoftheuniversity/grading/The content of these pages is in the static HTML (this is not a JS-rendered site), so
curl/ "save page as" is enough to inspect and to capture fixtures. Work out what distinguishes the real content (course descriptions, program requirements) from the catalog side-nav and the dense course cross-link lists — e.g. whether there's an inner container tighter than#main-content, and what class/element the nav menu uses. Let what you find drive the fix; don't guess selectors up front.It also helps to see the current output first: run the scraper's
scrape()against a couple of the existing (non-calendar) URLs fromdata/webpages/list.jsonand against one of the calendar URLs above, and compare. That shows you what good extraction looks like on the templates that already work, and makes the calendar noise concrete before you start changing anything.2. Update the extractor — gated to calendar hosts only
Keep all calendar-specific handling behind a hostname check so the other templates run exactly the code they run today. The extractor needs the page's URL to do that, so give
_extract_from_htmlapage_url: str = ""parameter and havescrape()pass the realurl. The default keeps existing callers — and the current tests — working unchanged.Branch on the host (e.g.
if "calendar.carleton.ca" in urlsplit(page_url).netloc:) and apply the fix inside that branch only. Based on the investigation, pick whichever fits the real markup (any combination):#main-contentto the inner content container the calendar template uses;Because everything is gated on the host, the CCSS and
carleton.capaths stay byte-for-byte identical to today — the shared single-selector scope and the<body>fallback are untouched. Still run the full scraper test to confirm no regression, since the branch lives in shared code.3. Add fixtures and tests
Capture two fixtures — the CS program page and the COMP courses page (the two noisy cases) — and save them under
tests/ingestion/fixtures/following the existing naming (e.g.calendarProgram.html,calendarCourses.html). The academic-year and regulations pages are for investigation/comparison; they don't need fixtures.Because the fix is gated on the host, the new test entries must pass the calendar URL into
_extract_from_html(html, url)so the branch actually fires — add aurlfield to thosepagesentries and pass it in the loop (the existing non-calendar entries have nourland default to"", so they call the extractor exactly as before).Extend the
pageslist intests/ingestion/test_scraper.pywith an entry per new fixture, asserting both sides:The current test uses a single shared
junklist checked against all pages. If a negative assertion only makes sense for the calendar pages, add a per-page exclude list rather than putting a calendar-specific string in the globaljunk(which would then be wrongly required absent from every other fixture too).📝 Notes
get_text(" ", strip=...)) still apply — follow the existing patterns.✅ Acceptance Criteria
_extract_from_htmlproduces clean content for calendar catalog pages: the CS program and COMP courses pages no longer include the catalog side-nav / menu, and the extracted text is dominated by real course/program content rather than nav links.page_url: str = ""parameter on_extract_from_html, passed through byscrape()), so the CCSS andcarleton.caoutput is unchanged and the existing fixtures pass with no edits to their assertions.tests/ingestion/fixtures/, with tests that pass the calendar URL in and assert real content is present and catalog-nav noise is absent.make testandmake lintpass.