Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
237 lines (201 loc) · 13.6 KB

ARCHITECTURE.md

File metadata and controls

237 lines (201 loc) · 13.6 KB
Raw

Technical Architecture

Overview

The PostgreSQL Engine addon replaces Statamic's flat-file Stache storage with PostgreSQL-native persistence. It integrates exclusively through Statamic's repository contracts — the same interfaces that Stache implements. This means all existing Statamic code (Query Builder, GraphQL, Antlers, Control Panel) works without modification.

The addon does not use Laravel Eloquent models for content persistence. All database operations use Laravel's query builder or raw SQL through dedicated storage services.

Activation

The addon activates when the POSTGRES_ENGINE_CONNECTION environment variable is set. Without it, the addon stays dormant and Statamic uses Stache as normal.

On activation, the ServiceProvider calls Statamic::repository() for each content type, overriding Stache's bindings with PostgreSQL-backed implementations. This happens during register(), before Statamic's own AppServiceProvider checks $this->app->bound($abstract).

Layer Architecture

┌─────────────────────────────────────────────────────────┐
│                    Statamic Runtime                       │
│  Entry::query()  Collection::all()  Antlers  GraphQL  CP │
└──────────────────────────┬──────────────────────────────┘
                           │ Statamic Repository Contracts
┌──────────────────────────▼──────────────────────────────┐
│                    Repositories                           │
│  EntryRepository  CollectionRepository  TaxonomyRepository│
│  TermRepository  GlobalRepository  NavigationRepository   │
│  AssetContainerRepository  FormRepository  etc.           │
└──────────────────────────┬──────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│                     Hydrators                             │
│  EntryHydrator  CollectionHydrator  TaxonomyHydrator      │
│  Convert DB rows ↔ native Statamic objects                │
└──────────────────────────┬──────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│                   Storage Services                        │
│  EntryStorage  CollectionStorage  TaxonomyStorage         │
│  Raw PostgreSQL operations via Laravel query builder      │
│  INSERT ... ON CONFLICT for upserts                       │
│  No Eloquent models                                       │
└──────────────────────────┬──────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│                     PostgreSQL                            │
│  pg_collections  pg_entries  pg_taxonomies  pg_terms       │
│  pg_globals  pg_navigations  pg_asset_containers          │
│  pg_assets_meta  pg_forms  pg_form_submissions            │
│  pg_revisions  pg_search_documents  pg_slow_queries       │
│  pg_entry_relationships  pg_entry_term_relationships      │
└─────────────────────────────────────────────────────────┘

Source Structure

src/
├── ServiceProvider.php          # Registers all bindings, validates config
├── Commands/
│   ├── DoctorCommand.php        # Health check
│   ├── ImportCommand.php        # Flat-file → PostgreSQL
│   ├── ExportCommand.php        # PostgreSQL → flat-file
│   ├── ReindexCommand.php       # Rebuild search indexes
│   ├── SearchTestCommand.php    # Interactive search testing
│   └── StatsCommand.php         # Query statistics
├── Contracts/
│   ├── ExportReport.php
│   ├── ExportService.php
│   ├── ImportReport.php
│   └── ImportService.php
├── Export/
│   ├── ContentExporter.php      # Exports all content types
│   └── ExportReportData.php     # Export report with counts/errors
├── Http/
│   └── Middleware/
│       ├── FlushIdentityMap.php         # Clears identity map after request
│       └── QueryProfilingMiddleware.php # Adds debug headers
├── Hydrators/
│   ├── AssetContainerHydrator.php
│   ├── CollectionHydrator.php
│   ├── EntryHydrator.php
│   ├── FormHydrator.php
│   ├── GlobalHydrator.php
│   ├── NavigationHydrator.php
│   ├── SubmissionHydrator.php
│   ├── TaxonomyHydrator.php
│   └── TermHydrator.php
├── Import/
│   ├── ContentImporter.php      # Imports all content types
│   └── ImportReportData.php     # Import report with counts/errors
├── Observability/
│   ├── QueryProfiler.php        # Request-scoped query tracking
│   └── SlowQueryLogger.php      # Persistent slow query logging
├── Query/
│   ├── EntryQueryBuilder.php    # Statamic Query Builder implementation
│   └── TermQueryBuilder.php     # Term query builder
├── Repositories/
│   ├── AssetContainerRepository.php
│   ├── CollectionRepository.php
│   ├── EntryRepository.php
│   ├── FormRepository.php
│   ├── GlobalRepository.php
│   ├── GlobalVariablesRepository.php
│   ├── NavigationRepository.php
│   ├── NavTreeRepository.php
│   ├── RevisionRepository.php
│   ├── SubmissionRepository.php
│   ├── TaxonomyRepository.php
│   └── TermRepository.php
├── Search/
│   ├── ContentNormaliser.php        # HTML/Markdown/Antlers stripping
│   ├── PostgresSearchEngine.php     # Full-text, fuzzy, hybrid search
│   └── SearchDocumentBuilder.php    # Builds search documents from content
├── Storage/
│   ├── AssetContainerStorage.php
│   ├── AssetMetaStorage.php
│   ├── CollectionStorage.php
│   ├── EntryStorage.php
│   ├── FormStorage.php
│   ├── FormSubmissionStorage.php
│   ├── GlobalStorage.php
│   ├── NavigationStorage.php
│   ├── RevisionStorage.php
│   ├── TaxonomyStorage.php
│   └── TermStorage.php
└── Support/
    ├── BulkLoader.php           # Batch loading for relationships
    ├── CycleDetector.php        # Prevents infinite recursion
    ├── IdentityMap.php          # Request-scoped object cache
    └── RelationshipIndexer.php  # Entry/term relationship indexing

Database Schema

Content Tables

Table Purpose Key columns
pg_collections Collection definitions handle (citext unique), title, sites (jsonb), route (jsonb), settings (jsonb)
pg_entries Entry content id (uuid), collection (citext), slug (citext), status, published, date, data (jsonb), soft-delete
pg_taxonomies Taxonomy definitions handle (citext unique), title, sites (jsonb), settings (jsonb)
pg_terms Taxonomy terms taxonomy (citext), slug (citext), data (jsonb), soft-delete
pg_globals Global variables handle (citext), site (citext), data (jsonb), unique on (handle, site)
pg_navigations Navigation trees handle (citext), site (citext), tree (jsonb), tree_path (metadata json)
pg_asset_containers Asset container config handle (citext unique), title, disk, settings (jsonb)
pg_assets_meta Asset metadata container (citext), path (citext), kind, mime_type, size, width, height, data (jsonb)
pg_forms Form definitions handle (citext unique), title, fields (jsonb), settings (jsonb)
pg_form_submissions Form submissions form_handle (citext), data (jsonb), ip_hash, user_agent_hash
pg_revisions Content revisions subject_type, subject_id (uuid), version (int), data (jsonb), unique on (type, id, version)

Derived/Index Tables

Table Purpose
pg_search_documents Full-text search index (tsvector, trigram)
pg_entry_relationships Entry-to-entry relationship index
pg_entry_term_relationships Entry-to-term relationship index
pg_slow_queries Slow query log for observability

PostgreSQL Features Used

Feature Purpose
UUID (gen_random_uuid()) Primary keys for all content
JSONB Flexible field data storage
citext Case-insensitive handles, slugs, collection names
tsvector Full-text search with language-aware stemming
pg_trgm (GIN) Fuzzy/trigram search for typo tolerance
unaccent Accent-insensitive search
Partial unique indexes Soft-delete support (unique slug per collection where not deleted)
ON CONFLICT DO UPDATE Idempotent upserts for import

Request Lifecycle

  1. Request arrives — Laravel routes to Statamic
  2. Statamic resolves repositories — Gets our PostgreSQL implementations via Statamic::repository() bindings
  3. Query Builder compiles SQL — EntryQueryBuilder translates Statamic's fluent API to PostgreSQL queries
  4. Storage executes query — EntryStorage runs the query via Laravel's query builder
  5. Hydrator creates objects — EntryHydrator converts DB rows to native Statamic\Entries\Entry objects
  6. Identity map caches — Same entry ID returns same PHP object instance within a request
  7. Statamic renders — Antlers, GraphQL, or CP uses the native objects as normal
  8. Middleware flushes — FlushIdentityMap clears the cache after the response

Import/Export Flow

Import (flat files → PostgreSQL)

content/collections/blog.yaml  →  CollectionStorage::insert()  →  pg_collections
content/collections/blog/*.md  →  EntryStorage::insert()       →  pg_entries
content/taxonomies/tags.yaml   →  TaxonomyStorage::insert()    →  pg_taxonomies
content/taxonomies/tags/*.yaml →  TermStorage::insert()         →  pg_terms
content/globals/*.yaml         →  GlobalStorage::insert()       →  pg_globals
content/navigation/*.yaml      →  NavigationStorage::upsert()   →  pg_navigations
content/assets/*.yaml          →  AssetContainerStorage::upsert()→ pg_asset_containers
resources/forms/*.yaml         →  FormStorage::upsert()         →  pg_forms
content/submissions/*/*.yaml   →  FormSubmissionStorage::insert()→ pg_form_submissions
content/revisions/*/*/*. yaml  →  RevisionStorage::insert()     →  pg_revisions

Each content type imports within its own transaction. Failures in one type don't affect others. Import is idempotent — uses PostgreSQL ON CONFLICT for upserts.

Export (PostgreSQL → flat files)

The reverse of import. Reads from storage services, writes YAML/Markdown files matching Statamic's expected directory structure. Excludes internal fields (id, created_at, updated_at) and sensitive metadata (ip_hash, user_agent_hash). Does not export observability data (search documents, slow queries, relationship indexes).

Search Architecture

The search system has three components:

  1. ContentNormaliser — Strips HTML, Markdown, and Antlers syntax from content. Removes accents for normalised text.
  2. SearchDocumentBuilder — Builds search documents from entries and terms. Applies configurable field weights (A/B/C/D). Calculates boost scores based on published status, recency, and custom fields.
  3. PostgresSearchEngine — Executes search queries in four modes:
    • Full-text: to_tsquery() against tsvector columns
    • Fuzzy: similarity() and % operator against trigram indexes
    • Hybrid: Combines full-text and fuzzy with weighted scoring
    • Accent-insensitive: Uses unaccent() function

Identity Map

The IdentityMap is a request-scoped cache that ensures the same database row always returns the same PHP object instance within a single request. This prevents duplicate hydration and ensures object identity (===) for the same entry.

The FlushIdentityMap middleware clears the map after each request to prevent stale data across requests.

Cycle Detection

The CycleDetector prevents infinite recursion when loading entries with circular relationships (A references B, B references A). It tracks the current load stack and returns false if a cycle is detected or the configured max depth is exceeded.

Hard Rules

These constraints are enforced by the test suite and cannot be violated:

  • No Eloquent models for content persistence
  • No statamic/eloquent-driver dependency — the ServiceProvider throws if detected
  • No custom required query API — everything works through standard Statamic methods
  • Preserve Statamic behaviour — Query Builder, GraphQL, Antlers, and CP must work identically to Stache
  • PostgreSQL-specific features are optional — the baseline works with standard PostgreSQL extensions only