Cli improvements + frontmatter - #2
Merged
Merged
Conversation
CLI now emits a fake-YAML frontmatter block by default for every fetch (status, final-url, title, description, fetched-at). The big motivator: many sites 302 to a /?err=404 home page and the previous output gave no signal that the original URL was dead. With final-url surfaced, those cases are now obvious. New flags: - --main: strip nav/header/footer/cookie/sidebar/related elements via injected JS before HTML extraction. Useful for cutting noise on noisy product/listing pages. - --head: print only the frontmatter block, skip the body (fast URL validation in batch). - --skip-frontmatter: backward-compat escape hatch for scripts that just want the body markdown. Implementation: - WebPageFetcher.fetch() returns a FetchedPage struct with html, statusCode, finalURL. Backwards-compatible fetchHTML() wrapper kept. - WKNavigationDelegate captures HTTP status via decidePolicyFor navigationResponse:; final URL read from webView.url at didFinish. - HTMLToMarkdown.extractMetadata() returns PageMetadata (title and description) using SwiftSoup; falls back from <title> to og:title and from meta[name=description] to og:description. - yamlEscape() in CLI quotes values containing colons, quotes, backslashes, comments, or leading/trailing whitespace; flattens newlines to spaces; escapes embedded quotes and backslashes. Tests: 8 new PageMetadataTests covering title/description extraction, OG fallbacks, whitespace trimming, empty-title fallback. All 54 tests pass.
Pick install destination by trying these in order, first writable wins: 1. $INSTALL_DIR (env override) 2. $(brew --prefix)/bin 3. /opt/homebrew/bin (Apple Silicon Homebrew default) 4. /usr/local/bin (Intel Homebrew / classic default; usually root-owned) Fixes the previous script always copying to /usr/local/bin/ which on Apple Silicon required sudo and didn't match where the existing binary lived. With Homebrew installed on Apple Silicon, the script now lands the binary at /opt/homebrew/bin/web-to-markdown and prints the chosen path. Fails with a clear error and the candidate list when nothing is writable.
Three new CLI flags for bridging the gap between page-load and SPA
hydration. All bounded by the existing --timeout.
--wait <seconds> Fixed delay after the page loads. Decimal
allowed (e.g. --wait 0.5). Default 0.
--wait-for <selector> Wait until at least one element matches the
CSS selector. Event-driven via MutationObserver,
no polling.
--wait-for-text <text> Wait until the body's visible text contains
this substring. Event-driven via MutationObserver.
The waits chain in order: load → fixed wait → wait-for selector →
wait-for-text → extract. Combine freely.
Implementation:
- Switched extraction from evaluateJavaScript to callAsyncJavaScript so
the wait sequence and the chrome-stripping (--main) live in one async
JS function with arguments passed safely (no string-escape gymnastics).
As a small bonus, the implicit microtask hop seems to give SPAs like
YouTube enough time to set document.title before extraction even
without an explicit --wait.
- The JS uses MutationObserver to detect selector/text appearance —
resolves on the first matching mutation, no polling loop.
- Outer Swift-side timeout (--timeout, default 30s) bounds all waits;
if the JS never resolves, the timeoutTask fails the fetch.
Tests: 4 new WebPageFetcherTests cover status/final-url capture,
wait-for matching immediately, wait-for timing out cleanly, and fixed
wait actually delaying extraction. All 58 tests pass.
Two ergonomics improvements for library consumers: 1. FetchOptions struct in WebToMarkdown: bundles the optional knobs so call sites don't grow with every new feature. New overload `WebPageFetcher.fetch(from:options:)` accepts it. Old positional overload kept for backward compat — it just builds a FetchOptions and forwards. 2. Frontmatter type (new file Frontmatter.swift): holds (page, metadata, fetchedAt) as raw Swift values, with a `format()` method that emits the YAML block the CLI produces. Consumers who want the data without formatting can read the stored properties directly; consumers who want the YAML can call format(). yamlEscape is also exposed as a public static for callers who want the same escape rules on their own values. CLI now uses both: builds a FetchOptions, calls `Frontmatter(page:, metadata:).format()`. Old private formatFrontmatter, yamlEscape, and truncate helpers in the CLI removed. Tests: 11 new FrontmatterTests covering the format output, value escaping (colons, quotes, backslashes, hashes, leading/trailing whitespace, empty values), newline flattening, plain-value passthrough, description truncation. All 69 tests pass. Also: README updated to use plain "YAML" (it parses as real YAML on the consumer side; the "fake" framing was misleading). Library example in README updated to show FetchOptions + Frontmatter usage.
There was a problem hiding this comment.
Pull request overview
This PR enhances the web-to-markdown CLI and library by adding richer fetch controls (wait conditions + “main content” extraction), returning response metadata (status + final URL), and emitting YAML frontmatter.
Changes:
- Extend
WebPageFetcherto return aFetchedPage(HTML + status code + final URL) and addFetchOptions(timeout, waits, chrome stripping). - Add CLI flags for frontmatter control and wait conditions, and document them in the README.
- Add
Frontmatter+PageMetadata(title/description extraction) with accompanying tests.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
bin/install.sh |
Improves install script UX by selecting a writable destination directory and printing clearer output. |
Sources/WebToMarkdown/WebPageFetcher.swift |
Adds FetchedPage, FetchOptions, wait/chrome-stripping behavior, and response metadata capture. |
Sources/WebToMarkdownCLI/WebToMarkdownCommand.swift |
Adds CLI flags/options for main extraction, waits, and frontmatter output modes. |
Sources/WebToMarkdown/HTMLToMarkdown.swift |
Introduces PageMetadata and metadata extraction from HTML. |
Sources/WebToMarkdown/Frontmatter.swift |
Adds YAML frontmatter rendering and escaping/truncation rules. |
Tests/WebToMarkdownTests/WebPageFetcherTests.swift |
Adds tests for status/final URL and wait behaviors. |
Tests/WebToMarkdownTests/HTMLToMarkdownTests.swift |
Adds unit tests for frontmatter formatting/escaping and metadata extraction. |
README.md |
Documents frontmatter output, new CLI flags, and updated library usage examples. |
Comments suppressed due to low confidence (1)
Sources/WebToMarkdown/WebPageFetcher.swift:279
Task.sleep(nanoseconds:)will trap iftimeoutis negative because the Double→UInt64 conversion is not valid. Add validation (e.g., precondition/guard thattimeout >= 0or clamp to 0) before converting to nanoseconds, ideally at the publicfetchAPI boundary.
timeoutTask = Task { @MainActor in
try? await Task.sleep(nanoseconds: UInt64(timeout * 1_000_000_000))
guard !Task.isCancelled,
state.access({ $0.fail(with: WebPageFetcher.Error.timeout) })
else { return }
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The decidePolicyFor navigationAction handler was logging every URL it saw, which on chrome-heavy pages produced a flood of about:blank / iframe / redirect entries in the Xcode console. The one-shot lifecycle events (Loading URL, Response status, Page loaded, Extracted HTML length) and all error paths are unaffected — they still always log. Adds FetchOptions.verbose (default false). The CLI's existing --verbose flag now passes through to it.
Contributor
|
@myobie A bunch of Copilot's suggestions make sense to me. |
…ge.finalURL as markdown base Two review fixes: - Throw a ValidationError if both --head and --skip-frontmatter are passed. Previously the combination silently produced empty output (frontmatter suppressed + early return after head). - Pass page.finalURL (not the raw input URL) as the base URL into HTMLToMarkdown.convert so relative links resolve correctly when the fetch followed redirects.
ISO8601 strings contain ':' so they need the same quoting as URLs and other colon-containing values. Previously the timestamp was emitted raw, producing technically-invalid YAML.
Frontmatter.format() quotes any value containing ':' (which both URLs and ISO timestamps do). The README example previously showed unquoted forms that didn't match real output.
Three test fixes: - waitForExistingSelectorReturnsImmediately: replace hardcoded 'elapsed < 5' with 'elapsed < timeout - 1' parameterized against the configured timeout, so a slow CI/network doesn't flake the test. - fixedWaitDelaysExtraction: loosen 'elapsed >= 1.0' to '>= 0.9' for ~100ms timer-precision/scheduling tolerance. - formatsAllFields: update expectation to the quoted form now that fetched-at runs through yamlEscape.
Member
Author
@atdrendel OK, I pushed the fixes I think. |
- Quote frontmatter values that start with YAML indicator characters - Observe attribute mutations when waiting for selectors - Capture status only for main-frame responses and use the latest final URL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sometimes it is helpful to see the status or wait for something specific.