fix: harden crawl filename handling and stale recovery

[austin-thesing]

Summary

• bound generated non-chunk asset filenames to prevent ENAMETOOLONG crashes, with regressions for long basenames, long extensions, and exact chunk-name preservation
• extracted worker runtime recovery settings, moved the stale recovery window to 10 minutes, and added WORKER_SKIP_LOCK_RENEWAL as an operator escape hatch
• included services/worker/src in the root test run and documented the worker recovery env vars in .env.example

Test plan

• bun test packages/scraper/src/asset-downloader.test.ts services/worker/src/worker-config.test.ts
• bun run verify

Summary by CodeRabbit

• New Features

  • Asset filenames are now bounded to a fixed max length while preserving distinguishing parts.
  • Worker runtime configuration introduced, exposing optional recovery/lock management settings.

• Documentation

  • Added commented env documentation for worker recovery, lock timing, reconciliation, and an opt-in lock-renewal flag.

• Tests

  • Expanded tests for asset filename edge cases and worker configuration/env parsing.

• Chores

  • Test runner updated to include worker service tests.

[railway-app]

🚅 Deployed to the dxd-webflow-scraper-pr-6 environment in WebFlow Archive

Service	Status	Web	Updated (UTC)
api	🕐 Queued (View Logs)	Web	Apr 6, 2026 at 6:56 pm
web	🕐 Queued (View Logs)	Web	Apr 6, 2026 at 6:56 pm
worker	🕐 Queued (View Logs)		Apr 6, 2026 at 6:56 pm

[coderabbitai]

📝 Walkthrough

Walkthrough

Centralizes worker runtime configuration into a new typed module, updates the worker processor to consume config-driven values (including orphan reconciliation and lock-renewal flags), bounds generated asset filenames to 200 chars with a new helper and tests, adds worker tests to the root test script, and documents new worker env vars in .env.example.

Changes

Cohort / File(s)	Summary
Worker Configuration services/worker/src/worker-config.ts, services/worker/src/worker-config.test.ts	Added WorkerRuntimeConfig type and getWorkerRuntimeConfig(env?) that parses positive-int and truthy boolean envs (defaults provided). New tests cover defaults and env overrides, including WORKER_SKIP_LOCK_RENEWAL behavior.
Worker Processor Integration services/worker/src/processor.ts	Replaced direct process.env parsing with getWorkerRuntimeConfig() usage. Flow updated to use config values for concurrency, lock duration, stalled interval, skipLockRenewal, and orphan reconciliation scheduling; reconcileOrphanedCrawls() now accepts orphanGraceMs.
Asset filename bounding & tests packages/scraper/src/asset-downloader.ts, packages/scraper/src/asset-downloader.test.ts	Introduced buildBoundedAssetFilename() to enforce a 200-char max filename while preserving hash/extension; added three tests for long basenames, long extensions, and existing chunk markers.
Env doc & test script .env.example, package.json	Added commented “Worker recovery” env vars to .env.example (WORKER_LOCK_DURATION_MS, ORPHAN_CRAWL_GRACE_MS, ORPHAN_CRAWL_RECONCILE_INTERVAL_MS, WORKER_SKIP_LOCK_RENEWAL) and kept VITE_API_URL as http://localhost:3001. Updated root package.json test script to include services/worker/src.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant Config as WorkerConfig
participant Processor as WorkerProcessor
participant Queue as BullMQ/Queue
participant Reconciler as OrphanReconciler
Config->>Processor: getWorkerRuntimeConfig()
Processor->>Queue: start workers with config.concurrency, config.lockDuration, config.skipLockRenewal
Processor->>Reconciler: schedule reconcile(orphanGraceMs) every config.reconcileIntervalMs
Reconciler->>Queue: find and requeue/mark orphaned crawls based on orphanGraceMs

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the two main changes: hardening crawl filename handling (asset downloader improvements) and stale recovery (worker config extraction with 10-minute window).

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/crawl-filename-stale-recovery

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.env.example:
- Around line 7-14: Add documentation for the ORPHAN_CRAWL_RECONCILE_INTERVAL_MS
environment variable in the "Worker recovery" section: describe that it controls
the reconcile interval for orphaned active crawls (milliseconds), note the
default value, show an example like
ORPHAN_CRAWL_RECONCILE_INTERVAL_MS=<default-ms>, and mention its relation to
ORPHAN_CRAWL_GRACE_MS and WORKER_LOCK_DURATION_MS so operators understand how
the timing knobs interact.

🪄 Autofix (Beta)

✅ Autofix completed

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 8a7ac387-2826-4036-aed8-779ce1d67923

📥 Commits

Reviewing files that changed from the base of the PR and between bef9d4d and 0b24f37.

📒 Files selected for processing (7)

• .env.example
• package.json
• packages/scraper/src/asset-downloader.test.ts
• packages/scraper/src/asset-downloader.ts
• services/worker/src/processor.ts
• services/worker/src/worker-config.test.ts
• services/worker/src/worker-config.ts

[coderabbitai]

Note

Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it.

Fixes Applied Successfully

Fixed 1 file(s) based on 1 unresolved review comment.

Files modified:

• .env.example

Commit: a71ba55079cf6df1a279ff11128727816db4aa43

The changes have been pushed to the fix/crawl-filename-stale-recovery branch.

Time taken: 3m 11s

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.env.example:
- Line 46: Add a trailing newline to the end of the .env.example file so the
file ends with a blank line (after the VITE_API_URL entry) to satisfy
dotenv-linter and standard POSIX conventions; simply ensure the last line
"VITE_API_URL=http://localhost:3001" is followed by a newline character.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: f7ce4c4b-99d1-494c-9115-d9d047917749

📥 Commits

Reviewing files that changed from the base of the PR and between 0b24f37 and a71ba55.

📒 Files selected for processing (1)

• .env.example

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Add blank line at end of file.

The dotenv-linter tool flags the missing trailing newline. Adding a blank line at the end follows standard convention and prevents potential issues with certain tools.

📝 Proposed fix

 VITE_API_URL=http://localhost:3001
+

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	VITE_API_URL=http://localhost:3001
	VITE_API_URL=http://localhost:3001

🧰 Tools

🪛 dotenv-linter (4.0.0)

[warning] 46-46: [EndingBlankLine] No blank line at the end of the file

(EndingBlankLine)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.env.example at line 46, Add a trailing newline to the end of the
.env.example file so the file ends with a blank line (after the VITE_API_URL
entry) to satisfy dotenv-linter and standard POSIX conventions; simply ensure
the last line "VITE_API_URL=http://localhost:3001" is followed by a newline
character.