Memory creep in production: WordPress containers steadily consume more memory (AWS ECS / EC2) - investigation required (--prod) -

[MartinPaulEve]

Summary

Production WordPress deployment is showing a steady increase in memory usage over time (memory creep), which appears to grow until the container is at risk of crashing. We are running on AWS ECS using EC2-backed containers.

Environment

• Repository: MESH-Research/knowledge-commons-wordpress
• Deployment: production (ECS, EC2 launch type)
• Service: wordpress-prod (multiple color instances: wp-green / wp-blue)

Observed behavior

• One of the WordPress production containers (wp-green) exhibits a gradual upward trend in memory utilization over multiple days.
• Periodic spikes are visible, but the baseline memory usage slowly increases over time (see evidence below).
• This behaviour suggests a memory leak or accumulation of in-memory state that is not released, ultimately increasing the risk of container OOM and crash.

Evidence

• CloudWatch/monitoring graph showing memory utilization for production containers (one-week window):

• The green series (wordpress-prod-3 wp-green) shows a clear upward trend from ~20% to ~40% (and higher on spikes) across the captured timeframe.
• The orange series (wp-cron) shows frequent short spikes (cron activity) but does not explain the persistent baseline increase.

Impact

• Production stability risk: containers may reach memory limits and be terminated / restarted by ECS, causing degraded performance or downtime for end users.
• Potential data-loss risk is low (stateless containers with persistent storage elsewhere) but restarts increase error rates and user-visible errors.

Suspected causes

• Long-running PHP processes (PHP-FPM) or a plugin/theme causing memory to be retained between requests.
• Improper object caching configuration (in-memory caches persisting beyond intended lifecycle).
• Background jobs or cron tasks (wp-cron) accumulating state or leaking memory on repeated execution.
• Native extension or third-party library (e.g., image processing, search indexing) leaking memory.

Immediate mitigation suggestions

• Temporarily reduce container memory reservation/limit or add an automated restart policy to recycle containers before they hit OOM (short-term stopgap).
• Scale out additional ECS tasks to reduce pressure on individual containers while investigating.
• Disable or roll back recent plugin/theme changes deployed shortly before the trend began (if applicable).

Recommended investigation steps

1. Collect more detailed runtime diagnostics from an affected container:
   • PHP-FPM status and process list (top/ps) over time.
   • per-process memory usage (e.g., ps aux --sort=-rss, pmap, smem) and heap snapshots where feasible.
   • PHP error logs and slow logs around spike times.
   • List of active plugins and recent changes/deploys.
2. Reproduce in a staging environment with high-traffic/certain cron schedules to see whether memory grows similarly.
3. Run memory-profiler on PHP (e.g., Xdebug/profiler, Blackfire, or other tools) for longer-running worker processes.
4. Check object-cache (Redis/Memcached) configuration and any in-memory object retention code paths.
5. Inspect background jobs / WP-Cron tasks for accumulating data or failing to free resources.
6. Correlate memory spikes with deployment events and cron spikes shown in the monitoring graph.

Priority

• High: production stability is affected and the issue can lead to container restarts and user-visible errors.

Attachments

• Monitoring graph included above.

[MartinPaulEve]

I've been through the full codebase (and the hcommons-mpe-theme repo, since it's deployed via EFS), the Docker/ECS/php-fpm configuration, and the pinned third-party stack looking for the source of this. Full write-up is now in MEMORY_LEAK.md at the repo root; summary and proposed plan below.

The short version: there's no single smoking-gun leak in our code. Almost every suspicious-looking code path turned out to be request-scoped — PHP-FPM frees that memory at request end, so it would produce a plateau, not a week-long linear climb. The evidence points instead at a small set of runtime conditions that the repo can't show because the values live in Secrets Manager or on EFS:

1. Worker recycling may be silently off. pm.max_requests = ${PM_MAX_REQUESTS} in config/all/php/www.conf has no fallback, and php-fpm 8.2 starts cleanly with pm.max_requests = 0 (never respawn) when the variable is unset — unlike pm.max_children, which refuses to start if missing. So the fact that prod runs proves the secret populates some of the PM family, but not that one. Never-recycled workers are the precondition that turns every small per-worker drift into unbounded growth.

2. Native-extension memory that memory_limit can't see. imagick is installed and is WordPress's preferred image editor; ImageMagick allocates outside PHP's memory manager, so leaked/fragmented native memory survives request shutdown and accumulates in long-lived workers. intl, yaml, protobuf, redis and memcached are also compiled in. This class of growth is exactly what pm.max_requests exists to contain.

3. OPcache is never enabled. The official php-fpm image compiles it but doesn't enable it, our php.ini replaces the stock one with 8 lines and no zend_extension, and it's not in the install-php-extensions list (I rebuilt the image to confirm — php -m shows no Zend OPcache). Every worker recompiles WP plus ~100 plugins on every request. Not itself a leak, but it inflates the baseline and amplifies 1 and 2.

4. The graph might partly be a measurement artifact. How the ECS agent reports memory depends on the host cgroup version: on cgroup v1 (AL2 AMI) the entire page cache is subtracted, so the creep would be real anonymous memory; on cgroup v2 (AL2023) only inactive file cache is subtracted, so continuously touched files (logs, /tmp, page cache from the EFS-mounted plugin tree) show up as monotonic "memory growth" that the kernel would happily reclaim before any OOM. One five-minute check on the container instance settles this and should be done first (commands in MEMORY_LEAK.md §5).

5. Nine active plugins/themes only exist on EFS and aren't in this repo or composer.json: elementor-pro, siteorigin-panels, 15zine-functionality, simplemag-addons, footnotes-made-easy, cron-logger, plus themes faculty, maskitto-light, origin, vantage-premium. None of them have been audited; any of them could contain the actual leak (Elementor already has documented high-memory issues upstream). We should pull them off /content and review them the same way.

6. Slow DB-side inflators. With the Redis object-cache drop-in install commented out in the entrypoint, every request reloads alloptions and the serialized cron option per site. Jetpack Sync has well-documented incidents of unbounded jpsq_sync-* rows in wp_options; the mla_academic_interests taxonomy grows forever from free-text profile input and gets fully hydrated on every autosuggest keystroke (its wp_cache_* guard never persists); ActivityPub 2.2 predates the outbox/inbox retention logic later versions added. KB-to-MB scale per request, but monotonic, and they compound with 1.

Things I chased and ruled out, so nobody re-treads them: the Monolog file logger in mu-plugins/logging.php is dead code (cbox-auth imports its own stderr-based fallback Logger, and nothing instantiates the Monolog class in either repo); nginx logs still go to stdout/stderr via the official image symlinks; the theme's header notifications shortcode is fine (BP 10.6 runs a grouped count query, not a full fetch); the BPGES digest queue is trimmed daily by the real crontab; bp-reply-by-email can't be in IMAP-daemon mode because the imap extension isn't in the image; and the cron task's sawtooth is just per-run WP-CLI processes exiting normally.

Proposed plan:

Diagnose first (read-only, detailed commands in MEMORY_LEAK.md §5): check the host cgroup version and the anon-vs-file split of the task cgroup (decides real leak vs metric artifact); dump the resolved FPM config with php-fpm -tt and cross-check the PM_*/PHP_* keys in the prod secret; snapshot ps worker ages now and 24h later; run the five DB-growth queries (autoloaded options size, biggest rows, jpsq_sync count, cron option length, interests term count); and tar up the EFS-only plugin set for review.

Then the config fixes, which are low-risk regardless of what the diagnosis says: set PM_MAX_REQUESTS=500 explicitly and have the entrypoint fail loudly if any of the PM family is missing; enable OPcache (validate_timestamps=0, max_accelerated_files sized for this codebase); enable the Redis object cache we already half-provision (redis ext, redis-cache plugin and WP_REDIS_* constants all exist — only the drop-in copy is commented out), with non-persistent connections to start; add per-container memory limits so growth becomes observable and attributable; set MAGICK_THREAD_LIMIT=1; and pin DISABLE_WP_CRON=true since the dedicated cron task already covers everything.

Code follow-ups (separate PRs): bound the academic-interests term query instead of loading the whole taxonomy per keystroke; upgrade BPGES past the batched-send rewrite so group posts stop fanning out emails synchronously in the posting request; cache elasticpress-buddypress's per-search _cat/indices cluster scan; clamp numberposts and add a permission callback on the public EPR REST endpoint; and upgrade or retire the vendored WPGraphQL 0.7.0 (no depth/complexity limits on a public batched endpoint).

If the cgroup check says the growth is real anon memory and the FPM values come back sane, the next suspects in order are the EFS plugin set and imagick accumulation — both covered in the runbook.