✨ Datafix Reconciliation Process

[BelSequent]

Parent issue: https://github.com/sequentech/meta/issues/12548

[Copilot]

Pull request overview

This PR introduces an external voter-registry reconciliation workflow (Datafix) end-to-end: admins can upload a reconciliation CSV, review the computed diffs (external-side + Sequent-side), download an external patch when needed, and apply Sequent-side changes via background tasks—while recording run-level audit entries in the electoral log. It also refactors existing “datafix” code into a more general services::external namespace and tightens some voted-channel/disable semantics.

Changes:

• Adds Windmill reconciliation generation/apply tasks plus supporting streaming diff/patch building, bulk Keycloak voter creation, and apply-time safety gates.
• Refactors Datafix integration to services::external and extends annotations/state tracking for reconciliation sequencing.
• Adds Hasura actions + Harvest routes + Admin Portal wizard UI, permissions, translations, and documentation for operating reconciliation.

Reviewed changes

Copilot reviewed 80 out of 85 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
REUSE.toml	Excludes generated Results Portal GraphQL artifacts from REUSE scanning.
packages/yarn.lock	Updates local sequent-core tarball resolutions.
packages/windmill/src/types/tasks.rs	Adds task execution enum entries for reconciliation generate/apply.
packages/windmill/src/tasks/process_cast_vote.rs	Migrates Datafix helpers/imports under services::external and updates lock key usage.
packages/windmill/src/tasks/mod.rs	Registers new reconciliation task modules.
packages/windmill/src/tasks/edit_user.rs	Refactors Datafix calls to services::external and clears stale voted-channel after discard.
packages/windmill/src/services/users.rs	Adds Keycloak DB-backed voter snapshot fetchers for reconciliation (paged + by-username).
packages/windmill/src/services/tasks_execution.rs	Adds update_with_annotations for structured task result annotations.
packages/windmill/src/services/mod.rs	Replaces datafix module export with external.
packages/windmill/src/services/insert_cast_vote.rs	Updates Datafix-related utilities import path and lock key function name.
packages/windmill/src/services/external/voterview_requests.rs	Updates imports to datafix_types module.
packages/windmill/src/services/external/utils.rs	Renames/extends Datafix utilities (lock key naming, channel comparisons, reconciliation sequence state).
packages/windmill/src/services/external/types.rs	Introduces reconciliation domain types (categories, patch target/source, Sequent-side field representation).
packages/windmill/src/services/external/reconciliation/mod.rs	Adds reconciliation module entrypoints.
packages/windmill/src/services/external/reconciliation/csv.rs	Implements reconciliation CSV/meta parsing with batched row iteration.
packages/windmill/src/services/external/reconciliation/patch.rs	Adds streaming writers for external patch CSV, diff JSON array, and NDJSON apply stream.
packages/windmill/src/services/external/reconciliation/bulk_create.rs	Bulk-creates missing voters directly in Keycloak DB for large-scale sync scenarios.
packages/windmill/src/services/external/reconciliation/apply.rs	Applies Sequent-side reconciliation changes per voter with guards and snapshot validation.
packages/windmill/src/services/external/mod.rs	Exposes new external modules (datafix_types, reconciliation).
packages/windmill/src/services/external/datafix_types.rs	Extends Datafix annotations with reconciliation apply state; adds reconciliation file/patch field types and channel mapping helpers.
packages/windmill/src/services/external/api_datafix.rs	Moves inbound Datafix API implementation under external and refines /unmark-voted behavior.
packages/windmill/src/services/electoral_log.rs	Adds a run-level electoral-log event for reconciliation generate/apply.
packages/windmill/src/services/celery_app.rs	Registers reconciliation tasks and routes them to Import/Export queue.
packages/windmill/src/services/cast_votes.rs	Updates Datafix utility import path.
packages/windmill/src/postgres/cast_vote.rs	Adds event-wide precomputed cast-vote state map for reconciliation.
packages/windmill/src/postgres/area.rs	Adds get_area_id_from_event_by_name for apply-time area-id resolution.
packages/windmill/src/tasks/generate_reconciliation_patches.rs	New task to compute diffs and upload envelope + patch artifacts.
packages/windmill/src/tasks/apply_reconciliation_patch.rs	New task to apply Sequent-side patch stream and persist structured result annotations.
packages/voting-portal/src/gql/graphql.ts	Regenerates GraphQL types to include reconciliation mutations.
packages/voting-portal/graphql.schema.json	Schema update including reconciliation mutations.
packages/step-cli/src/graphql/schema.json	Schema update including reconciliation mutations.
packages/sequent-core/src/types/permissions.rs	Adds election-event-voter-list-reconciliation permission constant.
packages/results-portal/src/gql/graphql.ts	Regenerates GraphQL types to include reconciliation mutations.
packages/results-portal/graphql.schema.json	Schema update including reconciliation mutations.
packages/harvest/src/routes/users.rs	Updates Datafix annotations import path.
packages/harvest/src/routes/mod.rs	Registers external reconciliation routes.
packages/harvest/src/routes/external_reconciliation.rs	Adds admin-authorized routes to start reconciliation generate/apply tasks.
packages/harvest/src/routes/error_catchers.rs	Updates Datafix error types import path.
packages/harvest/src/routes/api_datafix.rs	Updates inbound Datafix route wiring to services::external.
packages/harvest/src/main.rs	Mounts new external reconciliation routes.
packages/graphql.schema.json	Root schema update including reconciliation mutations.
packages/electoral-log/src/messages/statement.rs	Adds statement body/type/head formatting for reconciliation events.
packages/electoral-log/src/messages/newtypes.rs	Adds reconciliation kind + hash/sequence newtypes.
packages/electoral-log/src/messages/message.rs	Adds message constructor that can include an artifact payload for reconciliation apply.
packages/Dockerfile.keycloak	Installs findutils in the SPI build stage.
packages/ballot-verifier/src/gql/graphql.ts	Regenerates GraphQL types to include reconciliation mutations.
packages/ballot-verifier/graphql.schema.json	Schema update including reconciliation mutations.
packages/admin-portal/src/types/tasksExecution.ts	Adds reconciliation task enum values.
packages/admin-portal/src/types/keycloak.ts	Adds new permission + exports ATTR_RESET_VALUE constant.
packages/admin-portal/src/resources/VoterListSync/utils.ts	Adds helper to format reconciliation GeneratedAt timestamp.
packages/admin-portal/src/resources/VoterListSync/types.ts	Adds wizard types/enums matching backend reconciliation wire shapes.
packages/admin-portal/src/resources/VoterListSync/SyncDiffTable.tsx	Adds diff DataGrid component for reconciliation review tables.
packages/admin-portal/src/resources/VoterListSync/constants.ts	Adds category color mapping + highlighted categories set.
packages/admin-portal/src/resources/VoterListSync/CategorySummary.tsx	Adds category-count summary component for confirmation dialog.
packages/admin-portal/src/resources/User/useUsersPermissions.tsx	Adds permission gate for showing reconciliation UI.
packages/admin-portal/src/resources/User/ListUsers.tsx	Adds reconciliation wizard entry point in Voters tab and gates it by permission.
packages/admin-portal/src/queries/CreateExternalReconciliationImport.ts	Adds GraphQL mutation for starting diff generation.
packages/admin-portal/src/queries/ApplyExternalReconciliationChanges.ts	Adds GraphQL mutation for applying Sequent-side changes.
packages/admin-portal/src/gql/graphql.ts	Regenerates GraphQL types with reconciliation mutations.
packages/admin-portal/src/gql/gql.ts	Updates typed GraphQL document mapping for new mutations.
packages/admin-portal/graphql.schema.json	Schema update including reconciliation mutations.
packages/admin-portal/src/translations/en.ts	Adds reconciliation wizard translations (English).
packages/admin-portal/src/translations/es.ts	Adds reconciliation wizard translations (Spanish).
packages/admin-portal/src/translations/fr.ts	Adds reconciliation wizard translations (French).
packages/admin-portal/src/translations/nl.ts	Adds reconciliation wizard translations (Dutch).
packages/admin-portal/src/translations/gl.ts	Adds reconciliation wizard translations (Galician).
packages/admin-portal/src/translations/eu.ts	Adds reconciliation wizard translations (Basque).
packages/admin-portal/src/translations/cat.ts	Adds reconciliation wizard translations (Catalan).
packages/admin-portal/src/translations/tl.ts	Adds reconciliation wizard translations (Tagalog).
hasura/metadata/actions.yaml	Adds Hasura actions for reconciliation generate/apply with role permissions.
hasura/metadata/actions.graphql	Adds GraphQL action definitions for reconciliation generate/apply.
docs/permissions.md	Documents the new permission role.
docs/docusaurus/docs/02-election_managers/02-reference/02-election-event/05-election_management_election-event_voters.md	Adds operator-facing reconciliation runbook guidance.
CLAUDE.md	Updates dev guidance (avoid manual cargo build in dev container).
.devcontainer/keycloak/import/tenant-90505c8a-23a9-4cdf-a26b-4e19f6a097d5.json	Adds Keycloak role and assigns it in dev container import.
.devcontainer/docker-compose-base.yml	Passes KEYCLOAK_VOTER_GROUP_NAME into services.
.devcontainer/docker-compose-remote.yml	Passes KEYCLOAK_VOTER_GROUP_NAME into services.
.devcontainer/docker-compose-airgap-preparation.yml	Passes KEYCLOAK_VOTER_GROUP_NAME into services.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.