[pull] main from appwrite:main - #216
Merged
Merged
Conversation
Fix self-hosted 2.0 release blockers
appwrite/postgres:0.1.0 declares VOLUME /var/lib/postgresql and sets PGDATA=/var/lib/postgresql/18/docker, but the compose file mounted the named volume at /var/lib/postgresql/18/data - a sibling of PGDATA, not PGDATA itself. Docker therefore satisfied the declared VOLUME with a fresh anonymous volume on every container creation, and that anonymous volume is where all Postgres data actually lived. The named appwrite-postgresql volume stayed empty, so data survived a restart but was discarded by any container recreate: docker compose down, up --force-recreate, an image bump, or the documented upgrade entrypoint. Postgres is the default platform database, so a stock install lost its console users, projects and data on the first upgrade. Mount the named volume at /var/lib/postgresql so it covers PGDATA.
The upgrade task accepts --database but always re-derives the engine from the existing installation and locks it, so the flag was silently discarded. Someone running `upgrade --database=postgresql` on a 1.9.x MongoDB install got no indication that the request had been dropped and that their platform database was still MongoDB. Default the parameter to an empty string so an explicit value can be told apart from the default, and warn when it differs from the detected engine. The warning also states that Appwrite cannot move an existing installation between database engines, since there is no supported path for it. Behaviour is unchanged: the detected engine is still what the upgrade uses.
APP_CACHE_BUSTER was raised from 4326 to 4327 when this branch was prepared for 2.0.0, but 1.9.6 had already shipped 4327. The two releases therefore carried the same value. The constant is mixed into Request::cacheIdentifier(), so an installation upgrading from 1.9.6 kept serving responses cached under 1.9.6 keys instead of invalidating them. Bump to 4328 so 2.0.0 has a value no earlier release has used.
Two console schema changes made since 1.9.6 never reached upgraded installations. The boot-time sync in app/http.php only creates collections that do not exist yet, so it skips new attributes on collections that are already there, and V25 had no case for either one: - projects.onboarding, written by the API shutdown hook on every matching request and read by the console "Get started" checklist. Because the platform handle drops unknown attributes, the write was discarded silently and the checklist stayed empty forever. - schedules.projectInternalId and its two indexes, written by Projects/Http/Schedules/Create.php on every schedule create. Add both to V25 using the same guarded style as the surrounding cases, so re-running the migration warns instead of aborting. Verified by upgrading a seeded 1.9.6 install to 2.0.0 on MariaDB: both columns go from absent to present, seeded data survives, and a second migrate run is a no-op.
DocumentsDB is backed only by MongoDB and VectorsDB only by PostgreSQL, so an installation that uses either needs that engine reachable. The connection details were effectively hardcoded: - docker-compose.yml never listed the _APP_DB_*_DOCUMENTSDB variables, and listed no _APP_CONNECTIONS_DATABASE_* variables at all. Compose only forwards variables it names, so setting them in .env never reached the containers and both pools always fell back to mongodb:27017 and postgresql:5432. - Neither family was described in app/config/variables.php, so the installer never wrote them to .env either. - registers.php read _APP_DB_USER, _APP_DB_PASS and _APP_DB_SCHEMA for both pools, so the per-engine credential variables that compose did forward for VectorsDB were never consumed. Forward both families plus the two connection-string overrides on every service that resolves these pools, describe them in variables.php, and read the per-engine credentials with a fallback to the shared ones so existing configurations are unaffected. appwrite-worker-deletes and appwrite-worker-migrations resolve these pools too and previously received neither family; they now get both. appwrite-worker-stats-resources inherits them from appwrite-worker. Verified on a PostgreSQL installation with no mongodb service: pointing _APP_DB_HOST_DOCUMENTSDB at a MongoDB reachable under another name makes DocumentsDB create, write and read back, which was impossible before.
_APP_DB_PASS_DOCUMENTSDB and _APP_DB_PASS_VECTORSDB were declared with filter 'password'. The installer generates a value for any password variable that has no default, so a fresh install wrote a random secret to both. Those non-empty values then won the fallback in registers.php, while the bundled MongoDB and PostgreSQL services provision their user with _APP_DB_PASS, so both pools authenticated with a password the database never had. The generated values are also not sanitised for DSN use: the sanitiser in generatePasswordValue() only applies to names matching /^_APP_DB_.*_PASS$/, which these do not. Clear the filter so both variables install empty and fall back to _APP_DB_PASS, as the other pool overrides already do.
The dependency was declared through a hand-written package repository that pointed at the branch cursor/multi-job-queue-platform-f0c8 under the invented version 1.0.999. Nothing tagged contained the combined worker initialisation that app/worker.php relies on, so the build depended on a mutable branch reference that could be force-pushed or deleted. That work is merged into the monorepo main branch, and packages/platform there is identical to the pinned commit, so it has been tagged platform/1.0.0-rc18. Drop the package repository and depend on the tag. Only utopia-php/platform changes in the lock; every other package keeps its resolved version.
APP_LIMIT_DATABASE_BATCH was a hard-coded 100, so bulk row and document operations rejected anything larger with "Value must a valid array no longer than 100 items". On Cloud a plan raises it through databasesBatchSize, but self-hosted had no equivalent lever, leaving operators to split large loads into hundreds of sequential calls. Read the constant from _APP_LIMIT_DATABASE_BATCH, keeping 100 as the default so existing installations are unchanged, and forward it to the services that serve these endpoints. Every call site already reads $plan['databasesBatchSize'] ?? APP_LIMIT_DATABASE_BATCH, so the Cloud plan still takes precedence and only the fallback becomes configurable. The value is clamped to at least 1 so a malformed setting cannot disable bulk writes entirely. It is deliberately not capped at the top: the description warns that memory use and query size grow with it, and the operator owns that trade-off for their own hardware.
DocumentsDB runs only on MongoDB and VectorsDB only on PostgreSQL, but the installer provisioned a single engine, the one chosen for the platform. A default PostgreSQL installation therefore had no MongoDB, so creating a DocumentsDB database returned 201 and the first collection write failed with a bare server error, the cause visible only in the logs. A MariaDB installation lost both products the same way. Rather than always deploying every engine, ask which products the installation actually wants: - _APP_DOCUMENTSDB and _APP_VECTORSDB, both enabled by default, are asked during install, seeded from the environment for scripted runs, and preserved from the existing .env on upgrade. - The compose generator adds an engine only when the platform uses it or an enabled product needs it, so a deployment runs no database it has no use for. - A disabled product's routes throw GENERAL_SERVICE_DISABLED, for keys and privileged roles too, since the engine behind them is not deployed. Turning a product off now removes its engine instead of leaving an endpoint that fails on first write.
A product whose engine already backs the platform must not add a second one: MongoDB with DocumentsDB enabled is a single-engine deployment. The generator does this, but nothing asserted it, so a future change to how backing engines are resolved could start provisioning a duplicate without failing a test.
The CLI install asks whether to deploy DocumentsDB and VectorsDB, but the web installer had no equivalent, so anyone using it always got both engines with no way to opt out. Add a toggle for each alongside the database selector, carry the choice through the wizard state into the install request, and show both on the review step so the deployment is confirmed before it starts. The server maps them onto _APP_DOCUMENTSDB and _APP_VECTORSDB, which the compose generator already reads, so the engine set follows the same rules as the CLI path. Defaults come from the variable defaults, keeping both enabled unless an existing .env says otherwise.
Two problems with the database product toggles, both from review. The MongoDB support files were copied only when MongoDB backed the platform. With PostgreSQL or MariaDB selected and DocumentsDB enabled the generated compose still declares the mongodb service and bind-mounts mongo-init.js and mongo-entrypoint.sh, but neither file was written, so Docker created directories in their place and the container restarted in a loop. Copy them whenever MongoDB is required by the platform or by DocumentsDB. The web installer seeded both products as true in its form state, and setStateIfEmpty only fills empty values, so a rendered "disabled" default was ignored and applyStep1State re-checked the box. Upgrading an installation that had turned a product off silently submitted it as enabled. Seed both as null and hydrate them from the rendered defaults, as the other fields already do. Also drops an orphaned PHPDoc block left above getRequiredBackingServices() and renames $productEngines to $productToggles, since it maps a namespace to its environment variable rather than to an engine.
The release branch had fallen 55 commits behind main, so 2.0.0 would have shipped without the Open Runtimes orchestrator bump to 1.7.2, three OpenAPI spec fixes, FCM credential validation, dedicated scopes on the DocumentsDB and VectorsDB endpoints, the subquery limit constants and several lock fixes. Three files conflicted, all of them ones this branch had already touched: - app/config/variables.php and docker-compose.yml conflicted only because 2.0.x adds _APP_DOCUMENTSDB and _APP_VECTORSDB, which main does not carry yet. Both sides were additive, so ours is kept; _APP_LIMIT_DATABASE_BATCH already existed on both and is not duplicated. - composer.lock is taken from main, which is 55 commits newer, with the platform pin reapplied. It now differs from main by utopia-php/platform alone. Release metadata is unchanged: APP_CACHE_BUSTER stays 4328, APP_VERSION_STABLE stays 2.0.0, the README pins and console image tag are untouched.
Adds a Google slot to the console project's oAuthProviders, following the
same {provider}Enabled / {provider}Appid / {provider}Secret contract that
app/controllers/api/account.php reads at sign-in.
Without it createOAuth2Session('google') fails the Enabled check with
project_provider_disabled, which is what the console currently returns
even though the UI ships a Google button.
The fan-out grouped receivers by their matched subscription IDs so one encoded frame could serve many connections. Those IDs are ID::unique() per subscription, so no two connections can ever share one and the key never collapsed -- it built one group of one, every time. Replaced with a direct loop over $receivers, which says the same thing in fewer lines. Behaviour is identical: frames, message totals and byte totals are byte-for-byte equal across 1..1000 connections and 1..3 subscriptions per connection. Also meter outbound bytes. Fan-out cost is driven by payload size times fan-out width, not by message count, and nothing exposed that: during OnCall #1175 three pods stepped +18/+78/+226Mi at a message rate of 21/s, below a 74/s peek 70 minutes earlier that cost nothing. The quantity that actually moved was invisible. $outboundBytes was already computed for usage accounting; this puts it on a counter too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…upgrade refactor(install): versioned infrastructure migrations, carrying build artifacts across the 2.0 volume rename
Brings in the CDN certificate provider contract, which moves the certificates adapter out into utopia-php/cdn, and the fix that isolates parallel function archives -- the race behind the FunctionsSchedule and General e2e failures on this branch.
Realtime defaults to one Swoole worker per container instead of CPU count x _APP_WORKER_PER_CORE (6). Measured on a single worker holding ~1200 websocket connections: 0.06-0.35 cores, and ~84% of each additional worker's footprint is fixed per-process overhead rather than connection state (~37MiB fixed vs ~71KiB per connection). Extra workers also each subscribe to the firehose, so every worker json_decodes every event, and they split the accept distribution into a second balancing layer invisible to the deployment (connections per worker ranged 51-170 inside one container). Concurrency belongs to the deployment. _APP_WORKERS_NUM still overrides. Also serialise the event document once per event rather than once per subscriber. `subscriptions` is the only part of the frame that varies per connection and it is small, so the rest is encoded once and reused. A 199Hz profile taken during a live fan-out burst put this loop's json_encode at 26% of on-CPU work, against 2.8% at rest, with 257 of 285 samples on the pubsub callback's encode. Frames are unchanged apart from key order: 36 comparisons across unicode escaping, slashes, empty data, nested empties, a pre-set `subscriptions` key and a blob containing quotes and backslashes all decode identically. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inside `if ($total > 0)` at least one frame was sent, and every frame carries a non-empty literal envelope, so $outboundBytes cannot be zero. PHPStan proves it (greater.alwaysTrue) now that the frame is built by concatenation rather than json_encode, which could return false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Published SDK READMEs said they were compatible with server version X, which misled self-hosted users onto Cloud-ahead majors (e.g. Flutter 26 against Appwrite 1.9.6). Spell out that the target is Cloud and that self-hosted may need an older SDK release. Co-authored-by: chiragaggarwal5k <chiragaggarwal5k@gmail.com>
Reverts e92e56d, de7f258, 87c3467 and 90b4058, which asked in both installers which of DocumentsDB and VectorsDB to deploy and had the compose generator add an engine per enabled product. Neither product is ready to be turned on from a stock installation, so the choice is not worth its surface: two installer prompts, a web installer step, and per-product engine selection. This leaves 2.0.x matching main again for these files; the replacement, both products off behind an environment variable, lands on main in #13411 and reaches this branch through the usual sync.
…rsdb-2.0.x revert: drop the installer choice for the database products (2.0.x)
It only has to change when a response shape changes in a way a cached entry would get wrong. 2.0 does not, so leaving it at 4327 keeps caches warm through the upgrade rather than discarding every entry for nothing.
Every step render calls applyBodyDefaults(), which assigned the topology from the body dataset unconditionally. The dataset carries the value the page was first rendered with, so moving off step one put it back to combined and the install request carried combined however the radio was set. Picking "Separate" produced a combined install: one worker and one scheduler instead of a container per queue, with no sign anything had been ignored. It now seeds the same way as every other field, so the dataset supplies a starting value and a choice already made is left alone. The initial value moves to null for that -- setStateIfEmpty only fills what is empty, and 'combined' never was. (cherry picked from commit b52ded7)
…-values Normalize execution header response values
- Names the worker split "Container topology" rather than "Workers and schedulers", since other queues may join the same choice later, and drops the row that repeated it on the review screen. - Sizes the card to the step being shown. A single running maximum meant the tallest step set the floor for every other one, so the short ones carried its leftover height as dead space. Height now rides the same curve and frame as the panel cross-fade, and the outgoing panel holds its own height instead of being squeezed as the card resizes. - Field labels drop to 12px, leaving the section heading above them larger. - An SSL certificate email that was never entered reads as an "Empty" tag, matching how the other absent settings on that panel are shown. - The account step can be skipped. The installer already skips creating an account when either field is blank, so only the form was insisting; half an account is still refused. - HTTPS is on by default, since a public API is normally served over TLS. Local and plain HTTP installations have to turn it off.
Brings over DocumentsDB and VectorsDB shipping off by default, the canonical lowercase proxy rule domains, the certificate domain health check, and the orchestrator and CDN dependency bumps.
…arry a name - The HTTPS toggle follows the hostname: off for names no public certificate authority will issue for -- loopback names, .local and .internal, and bare IP literals -- and on for a real domain. Setting the toggle by hand stops it following, so an explicit choice stands. _APP_OPTIONS_FORCE_HTTPS keeps its own default; the toggle supplies the value the operator actually installs with. - The account step takes a name, which the server was deriving from the email. A blank one still derives. - The install endpoint accepts an account with both fields blank, matching the task it calls: createInitialAdminAccount() already skips creating one. Only the endpoint was insisting, so the step could not really be skipped. Either field filled still requires a valid pair. - Password managers stop offering to sign in on the setup step. They were reading the SSL certificate email revealed by the accordion as a login field; the fields that are not credentials for this site now opt out. The account step keeps its hints, so a manager can still save that one.
…terpolates it
Compose files before 2.0 write the image as
"${_APP_IMAGE:-appwrite/appwrite}:${_APP_VERSION:-latest}", and the tag is
everything after the first colon -- so the version read back was the rest of
that expression. It parses as no version at all, which counts as newer than
every release, so every infrastructure migration looked already applied and
none ran. A 1.9.x upgrade therefore never carried its build artifacts across,
which is the one thing those migrations exist to do.
A tag that does not start with a digit is now left to .env, which holds the
value the expression resolves to.
testProjectVariableInFunction created its function with 'timeout' => 15 and then executed it synchronously, asserting the execution completed. That timeout is not a handler budget. The executor spends it end to end: Docker.php bounds the container prepare wait, the launch wait and the cold-start TCP ping on it, decrementing the remainder after each stage, and only what survives is left for the function itself. A node-22 cold start on a contended runner consumes the lot, the executor raises RUNTIME_TIMEOUT, and the 500 turns the execution status into 'failed'. The budget was already almost exhausted on the happy path: a passing run spent 13.76s of the 15s, leaving 1.24s of margin, while a failing run reached 15.67s and logged "Timed out waiting for runtime." That is the ~20% failure rate this test shows in cloud CI, where it is the only file in the Project E2E job that creates a function at all. 60 matches setupDeployedFunction in FunctionsCustomServerTest, which was raised to 60 for the same reason. Assertions are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sactions-skipSubqueries
…nd docs The console sign-in credentials were only ever read in app/config/console.php, so they were absent from the three places every other _APP_* variable is declared: the committed .env, the appwrite service's environment pass-through, and the app/config/variables.php registry that documents them. Without the compose pass-through the variables never reach the container, and System::getEnv falls back to '' - which surfaces as a live sign-in button that fails with "provider disabled" and no other diagnostic. This covers GitHub, GitLab and Bitbucket as well as Google. Those three landed with the same gap and share the block being edited, so splitting them into a follow-up would touch these exact lines twice. Only the appwrite service declares them. The console document is built per request in app/init/resources.php and every read of the provider triple is on an HTTP path, so no worker or scheduler needs them - unlike _APP_VCS_*, which build workers do read. Defaults stay empty and filter stays '' deliberately: Install.php generates a random value for any empty var whose filter is 'token' or 'password', which would stamp junk into a fresh install's OAuth client secret.
…-coldstart-timeout fix(e2e): stop testProjectVariableInFunction failing on a node-22 cold start
feat(console): allow Google sign-in to the console project
Transactions subqueries SkipFilters
chore: track 2.0.x → main
docs(sdks): clarify Cloud vs self-hosted SDK compatibility
…ce IDs as booleans
Refactor presence handling to optimize memory usage by storing presen…
perf(realtime): single Swoole worker, encode the document once per event, meter outbound bytes
* fix(builds): cap the billed build duration at the build timeout Jobs::duration() measures wall clock from buildStartedAt, which the first log callback stamps — so a build that never streamed a line falls back to the deployment's creation time and measures its whole queue wait. That value is what Usage\Build::publish() bills, at memory x duration x cpus, and it is billed for 'failed' exactly as for 'ready'. The executor backend could not produce this: it measured microtime() around the build's own execution, so queue wait was structurally unbillable. Since builds moved to the jobs-service the mean billed duration of a *failed* build has gone 111s (Jun) -> 157s (Jul) -> 491s (Aug) while successful builds held at ~52s, and failed builds are now 57% of all billed build compute fleet-wide. One production project shows the shape plainly: 2,286 of its 2,311 failed August builds have buildStartedAt NULL, empty buildLogs, "exit code -1", and durations at 904-912s. On s-2vcpu-2gb that is ~1.0 GB-hour billed per build that never ran — 1,869 of its 1,878 billed GB-hours for the month. Clamp to _APP_COMPUTE_BUILD_TIMEOUT, the same ceiling Deployments hands the jobs-service as timeoutSeconds: no job outlives it, so nothing past it can have been build time. Guarded so a 0 or negative value leaves the measurement alone rather than zeroing every build. This is a ceiling, not the whole fix — a starved build still bills the full timeout. Billing 0 when buildStartedAt is NULL, and the build fan-out that starves them, are follow-ups. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(builds): drop the build duration cap test Removed at request; the cap in Jobs::duration() is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fix index scopes
…nshot testGetScreenshot rendered https://appwrite.io 53 times and asserted the resulting PNG was larger than 100000 bytes. That is an assertion about the marketing site's visual design, not about the endpoint. It went red across every cloud PR on 2026-08-31 when appwrite.io shipped an Init 2.0 splash screen: the test's cache-buster query makes the request a Cloudflare cache miss, waitUntil:load snapshots the splash before the page paints, and the cropped PNG is 10318 bytes. Feeding that render through Utopia\Image\Image::crop(800, 600) reproduces 10318 byte for byte off CI, so the failure is the page, not the runner's egress. Render example.com instead. It is IANA-reserved, carries no JavaScript, and renders bit-identically across repeated calls and cache-buster queries, which is why testGetScreenshotComparison already pixel-compares against it and stays green in the same lane. Assert the output the endpoint actually controls -- that the PNG comes back at the requested 800x600 -- and leave render fidelity to the pixel comparison that owns it. The endpoint's SSRF guard rules out an in-cluster target: PublicHostname rejects any host resolving into a private range and Domain::isKnown() requires a public suffix, so a hosts override pointing appwrite.io at the test stack would be refused before the browser is called. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…party-size test: stop asserting the weight of a third-party page in testGetScreenshot
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )