Fix engine state corruption on rollback timeout during initial boot#239
Merged
Conversation
When the engine worker initializes with an already installed application, initialize_version arms the rollback timer without setting deployment_to_terminate (that field is only set by full deployments). If the application failed to report running within the rollback timeout, the timeout handler restored the current instance with nil, and the next schedule tick crashed the engine worker when accessing the deployment state, leading to a supervisor restart loop. The rollback now falls back to an empty deployment when there is no previous deployment to return to, keeping the engine alive after ghosting the version. Risk assessment: - Impact: the engine worker survives a rollback of the very first deployment after a DeployEx restart. The version is still ghosted and the node is still stopped, as before. - Blast radius: the :timeout_rollback handler in Deployer.Engine.Worker. Rollbacks of regular full deployments keep the exact same behavior since deployment_to_terminate is always set on that path. - Regression risk: low. The new fallback only triggers where the code previously crashed. - Rollback: revert this commit.
thiagoesteves
added a commit
that referenced
this pull request
Jul 8, 2026
…guide (#244) * Update changelog with 0.9.5 entries and document changelog maintenance Add the missing 0.9.5 entries for the merged PRs: #239, #240, #241 and #242 as bug fixes, and #237 (CloudWatch log group retention) as an enhancement. Document the changelog maintenance rules in AGENTS.md so agents know how to add entries, and remove the outdated note that said the changelog must not be edited manually. Risk assessment: - Impact: documentation only, no runtime behavior changes. - Blast radius: CHANGELOG.md and AGENTS.md. - Regression risk: none, docs-only diff. - Rollback: revert this commit. * Updated changelog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the engine worker initializes with an already installed application, initialize_version arms the rollback timer without setting deployment_to_terminate (that field is only set by full deployments). If the application failed to report running within the rollback timeout, the timeout handler restored the current instance with nil, and the next schedule tick crashed the engine worker when accessing the deployment state, leading to a supervisor restart loop.
The rollback now falls back to an empty deployment when there is no previous deployment to return to, keeping the engine alive after ghosting the version.
Risk assessment:
Co-Authored-By: Claude Fable 5 noreply@anthropic.com