Architecture Refactor: SourceAdapter Pattern + SourceRegistry

[bradygaster]

Overview

Refactor ACCES discovery layer from monolithic content.ts/community.ts files into a pluggable SourceAdapter pattern with centralized SourceRegistry. This enables plug-and-play source extensibility, parallel execution, and graceful degradation. Prerequisite for all Phase 1 & Phase 2 source expansion work.

Background

Current architecture hardcodes discovery logic in two files (content.ts and community.ts), making it difficult to add new sources without tangling dependencies. Per gap analysis:

• Current state: ~32% signal completeness with RSS + GitHub only
• Architecture bottleneck: Cannot scale beyond 3-4 sources without significant code duplication
• Team decision (decisions.md): Implement adapter pattern before expanding sources

Requirements

• Extract RSS feed discovery into RSSSourceAdapter implementing SourceAdapter interface
• Extract GitHub repository discovery into GitHubSourceAdapter implementing SourceAdapter interface
• Create SourceAdapter interface with methods: \�alidate(), \discover(), \getName()\
• Create SourceRegistry class for auto-discovery, validation, and parallel execution
• Each adapter must own its own: env var requirements, rate limiting, error handling, validation logic
• Registry must support graceful degradation (skip broken adapters, log warnings)
• Maintain backward compatibility with existing ContentItem/DiscoveryResult types in src/types.ts
• No new external dependencies — use Node.js built-ins only

Squad SDK Integration

Use Squad SDK patterns for the adapter system:

• Adapter interface design: Follow Squad SDK's \SkillAdapter\ pattern (capability discovery, validation, execution)
• Registry pattern: Inspired by Squad SDK's skill registration system (auto-discovery, dependency injection)
• Error boundaries: Each adapter runs in isolation with Squad-style error handling (try/catch, log + continue)

Acceptance Criteria

• AC1: RSSSourceAdapter and GitHubSourceAdapter produce identical DiscoveryResult output to current implementation
• AC2: SourceRegistry.discoverAll() executes all adapters in parallel and returns combined results
• AC3: If one adapter fails, others continue execution (graceful degradation)
• AC4:
  pm run build && npm run start\ produces identical output files to current implementation (9 reports)
• AC5: TypeScript compiles with zero errors in strict mode
• AC6: New adapters can be added by creating a class implementing SourceAdapter and adding to registry — no changes to orchestrator (src/index.ts)

Dependencies

None — This is the foundation for all other source expansion work.

Assigned To

• McNulty (TypeScript implementation, testing)
• Stringer (Squad SDK architecture patterns, code review)

Estimated Effort

2.5 sessions

• Session 1: Define SourceAdapter interface, create SourceRegistry skeleton
• Session 2: Extract RSS + GitHub into adapters, test parity
• Session 3: Refine error handling, documentation, team review

[github-actions]

🏗️ Squad Triage — Freamon (Lead / Editor-in-Chief)

Issue: #1 — Architecture Refactor: SourceAdapter Pattern + SourceRegistry
Assigned to: Freamon (Lead / Editor-in-Chief)
Reason: No specific domain match — assigned to Lead for further analysis

@copilot evaluation: 🔴 Not suitable — issue involves work outside the coding agent's capability profile.

Team roster:

• Freamon (Lead / Editor-in-Chief) → label: squad:freamon
• McNulty (TypeScript Implementer) → label: squad:mcnulty
• Kima (Source Scout (Content)) → label: squad:kima
• Bunk (Source Scout (Community)) → label: squad:bunk
• Omar (Signal Analyst) → label: squad:omar
• Bubbles (Taxonomy + Dedupe Librarian) → label: squad:bubbles
• Daniels (Copilot SDK Specialist) → label: squad:daniels
• Stringer (Squad SDK Orchestrator) → label: squad:stringer
• Ralph (Work Monitor) → label: squad:ralph
• @copilot (Coding Agent) → label: squad:copilot

To reassign, remove the current squad:* label and add the correct one.

[github-actions]

🔄 Ralph — Auto-Triage

Assigned to: Freamon (Lead / Editor-in-Chief)
Reason: No domain match — routed to Lead

Ralph auto-triaged this issue via the squad heartbeat. To reassign, swap the squad:* label.

[bradygaster]

Implementation Plan: SourceAdapter Pattern + SourceRegistry

Author: McNulty (TypeScript Implementer)
Status: Ready for review

---

PRD Assessment

The PRD is solid. A few gaps I am filling in this plan:

1. Shared helpers — generateCanonicalId is duplicated in content.ts and taxonomy.ts; truncate is duplicated in content.ts and community.ts. Need a shared utils module.
2. Keyword constants — SEARCH_KEYWORDS, EXCLUSION_KEYWORDS, RSS_FEEDS, GITHUB_QUERIES should live inside their respective adapters (encapsulation), not in a shared config.
3. Skeleton sources — Blog search, YouTube, Reddit, social media skeletons get removed from the monolithic files. Future adapters will implement these as new SourceAdapter classes.
4. Logging — Adapters should return structured results; the registry handles console output. Adapters stay pure.
5. Error boundary — Each adapter catches its own errors (PRD says this). Registry wraps each discover() call in try/catch for defense-in-depth.

---

Architecture

src/
  types.ts                    # UNCHANGED
  discovery/
    adapter.ts              # NEW - SourceAdapter interface + SourceRegistry class
    adapters/
      rss.adapter.ts      # NEW - Extracted from content.ts
      github.adapter.ts   # NEW - Extracted from community.ts
    helpers.ts              # NEW - Shared helpers (generateCanonicalId, truncate, etc.)
    content.ts              # MODIFIED - thin re-export facade for backward compat
    community.ts            # MODIFIED - thin re-export facade for backward compat
  taxonomy.ts                 # MODIFIED - remove duplicate generateCanonicalId
  index.ts                    # MODIFIED - use SourceRegistry.discoverAll()
  analysis.ts                 # UNCHANGED
  state.ts                    # UNCHANGED
  output.ts                   # UNCHANGED

---

Exact Interface Definitions

SourceAdapter interface (src/discovery/adapter.ts)

import type { DiscoveryResult, RunState } from '../types.js';

export interface SourceAdapter {
  /** Human-readable name for logging */
  getName(): string;

  /**
   * Check whether this adapter can run.
   * Returns true if all prerequisites are met (env vars, network, etc.).
   * Must NOT throw - return false if not ready.
   */
  validate(): Promise<boolean>;

  /**
   * Execute discovery and return results.
   * May throw on transient errors - the registry handles graceful degradation.
   */
  discover(state: RunState): Promise<DiscoveryResult[]>;
}

AdapterResult type

export interface AdapterResult {
  adapter: string;
  results: DiscoveryResult[];
  error?: string;
  durationMs: number;
}

SourceRegistry class

export class SourceRegistry {
  private adapters: SourceAdapter[] = [];

  /** Register an adapter. Idempotent - skips if already registered by name. */
  register(adapter: SourceAdapter): void {
    if (this.adapters.some(a => a.getName() === adapter.getName())) return;
    this.adapters.push(adapter);
  }

  /** Return all registered adapter names. */
  getAdapterNames(): string[] {
    return this.adapters.map(a => a.getName());
  }

  /**
   * Validate and execute all registered adapters in parallel.
   * - Skips adapters that fail validation (logs warning).
   * - Catches errors from individual adapters (graceful degradation).
   * - Returns combined items + per-adapter metadata.
   */
  async discoverAll(state: RunState): Promise<{
    items: ContentItem[];
    adapterResults: AdapterResult[];
    errors: number;
  }> {
    // Phase 1: Validate all adapters in parallel
    const validationResults = await Promise.all(
      this.adapters.map(async (adapter) => ({
        adapter,
        valid: await adapter.validate(),
      }))
    );

    const validAdapters = validationResults.filter(r => r.valid).map(r => r.adapter);
    const skipped = validationResults.filter(r => !r.valid);
    for (const s of skipped) {
      console.log(`  Skipped ${s.adapter.getName()}: validation failed`);
    }

    // Phase 2: Execute all valid adapters in parallel
    const adapterResults: AdapterResult[] = await Promise.all(
      validAdapters.map(async (adapter) => {
        const start = Date.now();
        try {
          const results = await adapter.discover(state);
          return { adapter: adapter.getName(), results, durationMs: Date.now() - start };
        } catch (err) {
          const message = err instanceof Error ? err.message : String(err);
          return { adapter: adapter.getName(), results: [], error: message, durationMs: Date.now() - start };
        }
      })
    );

    // Phase 3: Aggregate
    const items = adapterResults.flatMap(r => r.results.flatMap(dr => dr.items));
    const errors = adapterResults.filter(r => r.error).length;
    return { items, adapterResults, errors };
  }
}

---

File-by-File Changes

1. CREATE src/discovery/helpers.ts

Extract shared utilities from content.ts and community.ts:

• generateCanonicalId() - single source of truth
• truncate()
• isAspireRelated()
• isExcluded()
• SEARCH_KEYWORDS constant
• EXCLUSION_KEYWORDS constant

2. CREATE src/discovery/adapter.ts

Contains SourceAdapter interface, AdapterResult type, and SourceRegistry class (shown above).

3. CREATE src/discovery/adapters/rss.adapter.ts

Extract from content.ts:

• RSS_FEEDS constant (stays private to this adapter)
• discoverFromRssFeeds() becomes RSSSourceAdapter.discover()
• inferContentType(), extractTopics() become private methods
• validate() always returns true (no env vars needed; RSS feeds are public)
• getName() returns "RSS Feeds"
• Imports helpers from ../helpers.js

4. CREATE src/discovery/adapters/github.adapter.ts

Extract from community.ts:

• GITHUB_QUERIES constant (stays private to this adapter)
• discoverFromGitHub() becomes GitHubSourceAdapter.discover()
• searchGitHubRepos(), searchGitHubIssues() become private methods
• inferGitHubContentType(), extractGitHubTopics(), inferIssueSignal() become private methods
• validate() returns true always (works without token at lower rate limit); logs warning if GITHUB_TOKEN not set
• getName() returns "GitHub Search"
• Imports helpers from ../helpers.js

5. MODIFY src/discovery/content.ts

Replace with thin backward-compat facade:

export { SEARCH_KEYWORDS, EXCLUSION_KEYWORDS, generateCanonicalId } from './helpers.js';
export async function discoverContent(state: RunState): Promise<DiscoveryResult[]> {
  const adapter = new RSSSourceAdapter();
  return adapter.discover(state);
}

Skeleton functions (discoverFromBlogSearch, discoverFromYouTube) removed - they returned [].

6. MODIFY src/discovery/community.ts

Replace with thin backward-compat facade:

export async function discoverCommunity(state: RunState): Promise<DiscoveryResult[]> {
  const adapter = new GitHubSourceAdapter();
  return adapter.discover(state);
}

Skeleton functions (discoverFromReddit, discoverFromSocialMedia) removed - they returned [].

7. MODIFY src/taxonomy.ts

• Remove the duplicate generateCanonicalId() function (lines 88-96)
• Add: export { generateCanonicalId } from './discovery/helpers.js';
• Preserves existing export surface

8. MODIFY src/index.ts

Replace the two try/catch discovery blocks with:

import { SourceRegistry } from './discovery/adapter.js';
import { RSSSourceAdapter } from './discovery/adapters/rss.adapter.js';
import { GitHubSourceAdapter } from './discovery/adapters/github.adapter.js';

// In main():
const registry = new SourceRegistry();
registry.register(new RSSSourceAdapter());
registry.register(new GitHubSourceAdapter());

const { items: allItems, adapterResults, errors: discoveryErrors } = await registry.discoverAll(state);
for (const result of adapterResults) {
  if (result.error) {
    console.error(`  Error ${result.adapter}: ${result.error}`);
  } else {
    const count = result.results.reduce((sum, r) => sum + r.items.length, 0);
    console.log(`  ${result.adapter}: ${count} items (${result.durationMs}ms)`);
  }
}

---

Migration Strategy

Phased approach - never break the build:

Step 1: Create new files (additive only) - Create helpers.ts, adapter.ts, adapters/rss.adapter.ts, adapters/github.adapter.ts. Run npm run build - must compile clean. These files are inert, nothing imports them yet.

Step 2: Wire up the registry in index.ts - Replace the two try/catch blocks with SourceRegistry.discoverAll(). Remove imports of discoverContent and discoverCommunity from index.ts. Run npm run build - must compile clean.

Step 3: Convert content.ts and community.ts to facades - Replace bodies with thin wrappers that delegate to adapters. This preserves backward compat for taxonomy.ts import of generateCanonicalId. Run npm run build - must compile clean.

Step 4: Clean up taxonomy.ts - Remove duplicate generateCanonicalId, re-export from helpers. Run npm run build - must compile clean.

Step 5: Verify parity - Run npm run start before and after, diff the output. Confirm identical ContentItem[] output.

---

Testing / Parity Verification

No test framework exists (PRD says no new dependencies), so verification is manual:

1. Snapshot before: Run npm run start on current code, save output folder
2. Snapshot after: Run npm run start on refactored code, save output folder
3. Diff: Compare JSON ledger files field-by-field (ignoring timestamps in source_first_seen)
4. Build gate: npm run build must produce zero errors in strict mode at every step
5. Smoke check: Each adapter individually returns the same shape as the original functions

---

What Gets Removed

• content.ts: discoverFromBlogSearch(), discoverFromYouTube() skeleton functions (returned [])
• community.ts: discoverFromReddit(), discoverFromSocialMedia() skeleton functions (returned [])
• taxonomy.ts: duplicate generateCanonicalId() (moved to helpers.ts)
• content.ts and community.ts duplicated truncate() (moved to helpers.ts)

Nothing functional is lost. The skeletons were empty. The duplicate helpers are consolidated.

---

Risks and Mitigations

Risk	Mitigation
RSS rate limiting under parallel execution	RSS feeds are fetched sequentially within the adapter; parallelism is between RSS and GitHub
GitHub API rate limits hit harder with parallel	GitHub adapter handles this internally; no change to rate behavior
Import path changes break downstream	Facades in content.ts/community.ts preserve all existing exports
generateCanonicalId divergence	Single source of truth in helpers.ts

---

Estimated Effort

• Step 1 (new files): 1 hour
• Step 2 (wire registry): 30 min
• Step 3 (facades): 30 min
• Step 4 (taxonomy cleanup): 15 min
• Step 5 (verify parity): 30 min
• Total: ~2.5-3 hours

---

Extension Point

After this refactor, adding a new source (e.g. Reddit) becomes:

// src/discovery/adapters/reddit.adapter.ts
export class RedditSourceAdapter implements SourceAdapter {
  getName() { return 'Reddit'; }
  async validate() { return true; }
  async discover(state: RunState): Promise<DiscoveryResult[]> { /* ... */ }
}

// src/index.ts - add 2 lines:
import { RedditSourceAdapter } from './discovery/adapters/reddit.adapter.js';
registry.register(new RedditSourceAdapter());

No other files change. AC6 satisfied.

[bradygaster]

Squad SDK Architecture Review — Stringer (Squad SDK Orchestrator)

Issue: #1 — Architecture Refactor: SourceAdapter Pattern + SourceRegistry
Reviewer: Stringer (Squad SDK Orchestrator)

---

Executive Summary

I've reviewed the PRD against the actual Squad SDK (@bradygaster/squad-sdk@0.8.24) internals. The SDK has three patterns directly relevant to this refactor, and the right integration level is "inspired-by, not coupled-to." Here's why and how.

---

1. Squad SDK Patterns That Map to This Work

Squad SDK Pattern	What It Does	ACCES Equivalent
SkillSource interface	Pluggable discovery from local/GitHub. Has listSkills(), getSkill(), getContent()	SourceAdapter with validate(), discover(), getName()
SkillSourceRegistry	Registers sources, resolves by priority, deduplicates	SourceRegistry with auto-discovery, parallel execution
ErrorFactory.wrap() + SquadError hierarchy	Wraps errors with severity, category, recoverability context	Adapter error boundaries with graceful degradation
EventBus	Pub/sub for lifecycle events with error isolation per handler	Pipeline stage events (optional, Phase 2)

---

2. Recommendation: Inspired-By, Not Coupled-To

Do NOT add @bradygaster/squad-sdk as a runtime dependency. Here's my reasoning:

1. The PRD says "no new external dependencies." Squad SDK brings @github/copilot-sdk and vscode-jsonrpc as transitive deps — that's heavyweight for a CLI content engine.
2. Squad SDK's SkillSource is about loading markdown knowledge files, not executing async I/O discovery pipelines. The shape is close but the semantics diverge.
3. What we actually need is the structural pattern, not the runtime. Mirror the contract, not the import.

The right level: Define SourceAdapter and SourceRegistry as standalone TypeScript interfaces that follow Squad SDK conventions. If we ever want to expose ACCES adapters as Squad skills (so agents can invoke discovery), we add a thin bridge later.

---

3. Proposed Interface Design (Squad-Aligned)

// src/discovery/source-adapter.ts

/** Mirrors Squad SDK's SkillSource pattern: name, type, validate, execute */
export interface SourceAdapter {
  /** Unique adapter identifier (kebab-case, matches Squad naming) */
  readonly name: string;
  /** Human-readable display name */
  readonly displayName: string;
  /** Discovery channel type */
  readonly channel: Channel;

  /** Squad-style capability check: can this adapter run? (env vars, auth, etc.) */
  validate(): Promise<AdapterValidation>;

  /** Execute discovery, return normalized results */
  discover(state: RunState): Promise<DiscoveryResult[]>;
}

export interface AdapterValidation {
  valid: boolean;
  reason?: string;  // Why validation failed
  warnings?: string[];  // Non-fatal issues
}

// src/discovery/source-registry.ts

/** Mirrors Squad SDK's SkillSourceRegistry: register, resolve, execute */
export class SourceRegistry {
  private adapters: Map<string, SourceAdapter> = new Map();

  /** Register an adapter (Squad-style register pattern) */
  register(adapter: SourceAdapter): void;

  /** Get all registered adapter names */
  list(): string[];

  /** Validate all adapters, return which are ready */
  validateAll(): Promise<Map<string, AdapterValidation>>;

  /**
   * Run all valid adapters in parallel with error isolation.
   * Squad-style: one adapter failure doesn't kill the pipeline.
   * Returns results from all successful adapters + error report.
   */
  discoverAll(state: RunState): Promise<RegistryResult>;
}

export interface RegistryResult {
  results: DiscoveryResult[];
  errors: Map<string, Error>;
  skipped: string[];  // Adapters that failed validation
  timing: Map<string, number>;  // Per-adapter execution time (ms)
}

---

4. Pipeline Mapping

The current pipeline in index.ts is: load state -> discover -> dedupe -> classify -> analyze -> output -> save state

How this maps to Squad SDK's execution model:

Pipeline Stage	Squad SDK Pattern	Implementation
Discover	SkillSourceRegistry.listAllSkills() — parallel resolve	SourceRegistry.discoverAll() — Promise.allSettled() with error isolation
Dedupe	N/A (Squad handles this at skill level)	Keep existing deduplicate() — no change needed
Classify	SkillRegistry.matchSkills() — scoring + ranking	Keep existing classify() — future: use Squad's scoring pattern for confidence
Analyze	N/A (domain-specific)	Keep existing analyze()
Output	N/A	Keep existing generateOutput()
Orchestration	EventBus.emit() per stage	Future: Wire up EventBus for pipeline observability

Key insight: Only the discovery layer benefits from the adapter pattern right now. The other stages are pure functions that don't need pluggability yet. Don't over-engineer.

---

5. Error Handling — Follow Squad's ErrorFactory Pattern

/** Squad-style error wrapping for adapter failures */
export class AdapterError extends Error {
  constructor(
    message: string,
    public readonly adapterName: string,
    public readonly recoverable: boolean,
    public readonly originalError?: Error,
  ) {
    super(`[${adapterName}] ${message}`);
  }
}

The registry's discoverAll() should use Promise.allSettled() — exactly how Squad's EventBus isolates handler errors. One adapter blowing up must never crash the pipeline.

---

6. Adapter Registration — Keep It Simple

// src/discovery/adapters/index.ts
import { SourceRegistry } from '../source-registry.js';
import { RSSSourceAdapter } from './rss.js';
import { GitHubSourceAdapter } from './github.js';

export function createDefaultRegistry(): SourceRegistry {
  const registry = new SourceRegistry();
  registry.register(new RSSSourceAdapter());
  registry.register(new GitHubSourceAdapter());
  return registry;
}

Don't do auto-discovery via filesystem scanning. Squad SDK's loadSkillsFromDirectory() pattern works for markdown files, but for TypeScript adapters we want explicit registration. It's safer, tree-shakeable, and the import graph documents what's active.

---

7. What Changes in index.ts

The orchestrator (src/index.ts) changes from:

// Before — hardcoded calls
const contentResults = await discoverContent(state);
const communityResults = await discoverCommunity(state);

To:

// After — registry-driven
const registry = createDefaultRegistry();
const validation = await registry.validateAll();
// Log which adapters are active
const { results, errors, skipped } = await registry.discoverAll(state);
// Log errors and skipped adapters

AC6 is satisfied: Adding a new adapter = create class + add registry.register() call. Zero changes to orchestrator.

---

8. Future Bridge to Squad SDK (Phase 2+)

If we want adapters to be invocable as Squad skills (so agents can trigger discovery runs), the bridge is trivial (~10 LOC). Don't build it now. Build it when we actually need agent-invoked discovery.

---

Summary of Recommendations

1. Pattern: Mirror SkillSource/SkillSourceRegistry contracts, don't import them
2. No new deps: Keep @bradygaster/squad-sdk out of package.json runtime deps
3. Error handling: Follow ErrorFactory.wrap() + Promise.allSettled() isolation
4. Registration: Explicit register() calls, no filesystem scanning
5. Scope: Only refactor discovery layer. Other pipeline stages stay as-is
6. Future bridge: Design the interface so a Squad SDK bridge is trivial later (~10 LOC)
7. Naming: Use Squad-style kebab-case identifiers (rss-feeds, github-repos)
8. Validation: Each adapter owns its env var checks (like Squad's capability discovery)

---

— Stringer, Squad SDK Orchestrator
"The orchestration layer is what turns a collection of scripts into a real product."