diff --git a/docs/docusaurus/docs/02-election_managers/02-reference/02-election-event/08-03-election_management_election-event_tally-sheet-imports.md b/docs/docusaurus/docs/02-election_managers/02-reference/02-election-event/08-03-election_management_election-event_tally-sheet-imports.md index c3e54dcf3c7..eb8e0beaf94 100644 --- a/docs/docusaurus/docs/02-election_managers/02-reference/02-election-event/08-03-election_management_election-event_tally-sheet-imports.md +++ b/docs/docusaurus/docs/02-election_managers/02-reference/02-election-event/08-03-election_management_election-event_tally-sheet-imports.md @@ -46,7 +46,7 @@ channel + area_name + contest_external_id The selected import channel must match every `channel` value in the CSV. -Each ballot box must include exactly one row for every scalar field: `total_votes`, `total_valid_votes`, `implicit_invalid`, `explicit_invalid`, `total_blank_votes`, and `census`. It must also include one `candidate_votes` row for every candidate in the matched STEP contest. +Each ballot box must include exactly one row for every required scalar field: `total_votes`, `total_valid_votes`, `implicit_invalid`, `explicit_invalid`, `total_blank_votes`, and `census`. It must also include one `candidate_votes` row for every candidate in the matched STEP contest. Any other `field` value (e.g. `over_votes`/`under_votes`, written by the ES&S import path — see below) is accepted without validation and carried through as extra data on the imported ballot box, so new source-specific fields don't need a canonical CSV format change to be picked up. Uploaded files are stored as normal STEP documents. The UI and CLI send a SHA-256 hash of the source file when they upload it, and the import actions verify that hash before parsing or persisting the import. The import record stores both the original source hash and the canonical CSV hash used for validation and review. @@ -58,7 +58,7 @@ Uploaded files are stored as normal STEP documents. The UI and CLI send a SHA-25 `contest_external_id` must match the contest external ID configured in STEP. Because the canonical CSV format has no election column, **contest external IDs must be unique within an election event** for tally sheet import to resolve rows correctly. If an election event has multiple elections that reuse the same external ID for different contests, the import will fail with an ambiguous-match error — rename the duplicate external IDs before importing. -`field` must be one of: +`field` is usually one of the required scalar fields or `candidate_votes`: - `candidate_votes` - `total_blank_votes` @@ -68,6 +68,8 @@ Uploaded files are stored as normal STEP documents. The UI and CLI send a SHA-25 - `total_votes` - `census` +Any other value (e.g. `over_votes`, `under_votes`) is accepted as unvalidated extra data — see the note above. + `candidate_external_id` is required only when `field` is `candidate_votes`. It must match a candidate external ID in the matched contest. `value` must be a non-negative integer. @@ -86,11 +88,14 @@ The importer validates these invariants per ballot box: ```text total_invalid = implicit_invalid + explicit_invalid -total_valid_votes = sum(candidate_votes) + total_blank_votes +non_blank_valid_votes = total_valid_votes - total_blank_votes +non_blank_valid_votes <= sum(candidate_votes) <= non_blank_valid_votes * max_marks_per_ballot total_votes = total_valid_votes + total_invalid total_votes <= census ``` +A voter may be allowed to mark more than one candidate per ballot, so `sum(candidate_votes)` isn't required to equal the ballot count — it only has to fall within the range a valid ballot for this contest could produce. `max_marks_per_ballot` is `1` for ordinary single-choice contests, the contest's `max_votes` for plurality-at-large "vote for N" contests, and `max_votes` multiplied by the cumulative-voting checkbox limit for cumulative contests. + ## ES&S Enhanced XML Mapping For ES&S Enhanced XML uploads: @@ -99,9 +104,12 @@ For ES&S Enhanced XML uploads: - contest `altId1` is matched to contest external ID - candidate `altId1` is matched to candidate external ID - only ES&S reporting group `1` is read: its values are summed per precinct into one result for the selected STEP channel, and all other reporting groups are ignored -- ES&S overvotes become STEP implicit invalid votes -- ES&S blank votes become STEP blank votes -- ES&S undervotes are treated with the overlap-safe rule `max(underVotes - blankVotes, 0)` before adding them to implicit invalid votes +- ES&S overvotes always become STEP implicit invalid votes. ES&S's `overVotes` is a selection-*slot* count, not a ballot count — an overvoted ballot always contributes its whole `max_votes` allotment to `overVotes` (confirmed by the EVS SOP and empirically). For contests where ballots-cast is reported per contest and precinct (the `ContestReportingGroupVotes` XML variant), `overVotes` is divided by `max_votes` to recover an overvoted-*ballot* count before it's added to `implicit_invalid`; otherwise it could exceed `total_votes` and make `total_valid_votes` underflow. The other ES&S variant derives `total_votes` from candidate marks/blank votes instead (see below), so it isn't exposed to that underflow and uses the raw `overVotes` count directly. +- ES&S undervotes only become implicit invalid votes (via the overlap-safe rule `max(underVotes - blankVotes, 0)`) when the contest requires a minimum number of selections (`min_votes > 0`) — undervoting an otherwise-optional contest is never invalid +- ES&S's own `blankVotes` figure is **not** used as STEP's `total_blank_votes` for contests where ballots-cast is reported per contest and precinct (the `ContestReportingGroupVotes` XML variant): per the ES&S EVS documentation, `blankVotes` is exactly `overVotes + underVotes`, not a genuine blank-ballot count, so it can't validate a "vote for N" contest correctly. `underVotes` alone is used instead, which reconciles exactly for single-choice contests. The other ES&S variant (candidate-reporting-group) doesn't have this problem — its blank figure comes from the precinct's own `blanksCast`, a genuine (if precinct-wide) blank-ballot count — so it keeps using that. +- `total_votes`/`total_valid_votes` are derived from ES&S's own ballots-cast figure when it's reported per contest *and* precinct (the `ContestReportingGroupVotes` XML variant), so vote-for-N contests (where candidate marks legitimately exceed the ballot count) convert correctly. The other ES&S variant only reports ballots cast per precinct, shared across every contest on the ballot — not a valid ballot count for a contest that doesn't appear on every ballot style in that precinct (e.g. a ward- or school-board-specific race). For that variant, `total_valid_votes` is derived from candidate marks plus blank votes instead, the only figures actually scoped to that contest and precinct; `census` always uses the precinct-wide ballots cast regardless of variant. +- for the `ContestReportingGroupVotes` variant, the import additionally writes `over_votes`/`under_votes` rows carrying ES&S's raw counts, and cross-checks the per-ballot accounting identity documented in the ES&S SOP: every one of a contest's `max_votes` selection slots, across every ballot cast for it, is either a candidate mark, an overvote slot, or an undervote slot, with no remainder (`sum(candidate_votes) + over_votes + under_votes == total_votes * max_votes`). A mismatch is reported as an `ess_vote_reconciliation_mismatch` validation error — it indicates a data-quality problem in the source file, not something STEP can correct on its own. +- for the `ContestReportingGroupVotes` variant, the import also checks that `over_votes` is an exact multiple of `max_votes` — required for the overvoted-ballot count used to compute `implicit_invalid` (see above) to be well-defined. A non-exact remainder is reported as an `ess_over_votes_not_divisible` validation error. ## Import Lifecycle diff --git a/packages/admin-portal/rust/sequent-core-0.1.0.tgz b/packages/admin-portal/rust/sequent-core-0.1.0.tgz index 1912c3461cb..74a824cf49f 100644 Binary files a/packages/admin-portal/rust/sequent-core-0.1.0.tgz and b/packages/admin-portal/rust/sequent-core-0.1.0.tgz differ diff --git a/packages/admin-portal/src/resources/TallySheet/EditTallySheet.tsx b/packages/admin-portal/src/resources/TallySheet/EditTallySheet.tsx index e0c2c33999b..26f4ad9edd9 100644 --- a/packages/admin-portal/src/resources/TallySheet/EditTallySheet.tsx +++ b/packages/admin-portal/src/resources/TallySheet/EditTallySheet.tsx @@ -31,7 +31,7 @@ import { } from "@mui/material" import TextField from "@mui/material/TextField" import {IAreaContestResults, ICandidateResults, IInvalidVotes} from "@/types/TallySheets" -import {sortFunction} from "./utils" +import {sortFunction, translateSharedValidationError} from "./utils" import { EEnableCheckableLists, ICandidatePresentation, @@ -86,10 +86,19 @@ interface SharedValidationError { code: string message: string field: string + params: Record } -const validateAreaContestResults = (content: IAreaContestResults): SharedValidationError[] => - validate_area_contest_results_js(content) +interface IContestMarkBounds { + max_votes?: Maybe + counting_algorithm?: Maybe + cumulative_number_of_checkboxes?: number +} + +const validateAreaContestResults = ( + content: IAreaContestResults, + contestBounds: IContestMarkBounds +): SharedValidationError[] => validate_area_contest_results_js(content, contestBounds) const numbersRegExp = /^[0-9]+$/ @@ -107,6 +116,9 @@ export const EditTallySheet: React.FC = (props) => { const {t, i18n} = useTranslation() const aliasRenderer = useAliasRenderer() + const translateValidationError = (error: SharedValidationError): string => + translateSharedValidationError(t, error) + const [areasList, setAreasList] = useState([]) const [contestList, setContestList] = useState([]) const [channel, setChannel] = React.useState(null) @@ -125,9 +137,9 @@ export const EditTallySheet: React.FC = (props) => { const [areaNameFilter, setAreaNameFilter] = useState(null) const [contestNameFilter, setContestNameFilter] = useState(null) const [areaIds, setAreaIds] = useState>([]) - const [totalValidError, setTotalValidError] = useState(false) - const [censusError, setCensusError] = useState(false) - const [sharedValidationMessages, setSharedValidationMessages] = useState([]) + const [sharedValidationErrors, setSharedValidationErrors] = useState( + [] + ) const {data: areaContests} = useGetList( "sequent_backend_area_contest", { @@ -375,24 +387,22 @@ export const EditTallySheet: React.FC = (props) => { } } - const sharedValidationErrors = validateAreaContestResults({ - ...newResults, - invalid_votes: invalids, - candidate_results: candidateResultsForValidation, - }) - - const codes = new Set(sharedValidationErrors.map((error) => error.code)) - setTotalValidError(codes.has("invalid_total_valid_votes")) - setCensusError(codes.has("total_votes_exceeds_census")) - setSharedValidationMessages( - sharedValidationErrors - .filter( - (error) => - error.code !== "invalid_total_valid_votes" && - error.code !== "total_votes_exceeds_census" - ) - .map((error) => error.message) + const contestPresentation = choosenContest?.presentation as IContestPresentation | undefined + const sharedValidationErrors = validateAreaContestResults( + { + ...newResults, + invalid_votes: invalids, + candidate_results: candidateResultsForValidation, + }, + { + max_votes: choosenContest?.max_votes, + counting_algorithm: choosenContest?.counting_algorithm, + cumulative_number_of_checkboxes: + contestPresentation?.cumulative_number_of_checkboxes, + } ) + + setSharedValidationErrors(sharedValidationErrors) setIsButtonDisabled(sharedValidationErrors.length > 0) if (JSON.stringify(newResults) !== JSON.stringify(results)) { @@ -407,6 +417,7 @@ export const EditTallySheet: React.FC = (props) => { results.total_valid_votes, invalids?.total_invalid, invalids, + choosenContest, ]) const handleChange = ( @@ -588,6 +599,18 @@ export const EditTallySheet: React.FC = (props) => { } }, [choosenContest]) + const totalValidVotesError = sharedValidationErrors.find( + (error) => error.code === "invalid_total_valid_votes" + ) + const censusExceededError = sharedValidationErrors.find( + (error) => error.code === "total_votes_exceeds_census" + ) + const otherValidationErrors = sharedValidationErrors.filter( + (error) => + error.code !== "invalid_total_valid_votes" && + error.code !== "total_votes_exceeds_census" + ) + return ( <> @@ -693,13 +716,13 @@ export const EditTallySheet: React.FC = (props) => { size="small" required /> - {totalValidError && ( - - {t("tallysheet.inputError.totalValidDoesNotMatch")} - + {totalValidVotesError && ( + {translateValidationError(totalValidVotesError)} )} - {sharedValidationMessages.map((message) => ( - {message} + {otherValidationErrors.map((error) => ( + + {translateValidationError(error)} + ))} = (props) => { size="small" required /> - {censusError && ( - {t("tallysheet.inputError.censusTooSmall")} + {censusExceededError && ( + {translateValidationError(censusExceededError)} )} diff --git a/packages/admin-portal/src/resources/TallySheet/utils/index.ts b/packages/admin-portal/src/resources/TallySheet/utils/index.ts index 2e5473eb876..3bed351ad41 100644 --- a/packages/admin-portal/src/resources/TallySheet/utils/index.ts +++ b/packages/admin-portal/src/resources/TallySheet/utils/index.ts @@ -3,6 +3,7 @@ // SPDX-License-Identifier: AGPL-3.0-only import {isString} from "@sequentech/ui-core" +import {useTranslation} from "react-i18next" export const sortFunction = (a: {name?: string | null}, b: {name?: string | null}) => { if (isString(a?.name) && isString(b?.name)) { @@ -10,3 +11,38 @@ export const sortFunction = (a: {name?: string | null}, b: {name?: string | null } return 0 } + +export interface ISharedValidationError { + code: string + message: string + field?: string | null + params?: Record | null +} + +// Maps a shared tally sheet validation error's `code` (from +// sequent_core::services::tally_sheet_validation, used by both manual entry +// and tally sheet import) to the i18n key used to render it. Codes without +// an entry here fall back to `tallysheet.inputError.`. +const VALIDATION_ERROR_TRANSLATION_KEYS: Record = { + invalid_total_valid_votes: "tallysheet.inputError.totalValidDoesNotMatch", + total_votes_exceeds_census: "tallysheet.inputError.censusTooSmall", + invalid_total_invalid: "tallysheet.inputError.totalInvalidDoesNotMatch", + invalid_total_votes: "tallysheet.inputError.totalVotesDoesNotMatch", +} + +// `t` is typed loosely (matching react-i18next's `useTranslation().t`) +// rather than importing i18next's TFunction, to avoid a new type +// dependency for a single helper. +export const translateSharedValidationError = ( + t: ReturnType["t"], + error: ISharedValidationError +): string => { + const translationKey = + VALIDATION_ERROR_TRANSLATION_KEYS[error.code] ?? `tallysheet.inputError.${error.code}` + return String( + t(translationKey, { + ...(error.params ?? {}), + defaultValue: error.message, + }) + ) +} diff --git a/packages/admin-portal/src/resources/TallySheetImport/TallySheetImports.tsx b/packages/admin-portal/src/resources/TallySheetImport/TallySheetImports.tsx index 4da8ac9bb4a..079c86ef740 100644 --- a/packages/admin-portal/src/resources/TallySheetImport/TallySheetImports.tsx +++ b/packages/admin-portal/src/resources/TallySheetImport/TallySheetImports.tsx @@ -48,6 +48,7 @@ import { useRefresh, } from "react-admin" import {useTranslation} from "react-i18next" +import {translateSharedValidationError} from "@/resources/TallySheet/utils" import ElectionHeader from "@/components/ElectionHeader" import {ListActions} from "@/components/ListActions" import {GET_UPLOAD_URL} from "@/queries/GetUploadUrl" @@ -101,6 +102,7 @@ interface TallySheetImportValidationError { contest_external_id?: string | null candidate_external_id?: string | null field?: string | null + params?: Record | null } interface TallySheetImportPreviewItem { @@ -1229,17 +1231,20 @@ const SummaryChip: React.FC<{label: string; value: number}> = ({label, value}) = ) -const ValidationErrors: React.FC<{errors: TallySheetImportValidationError[]}> = ({errors}) => ( - - - {errors.map((error, index) => ( - - {error.message} - - ))} - - -) +const ValidationErrors: React.FC<{errors: TallySheetImportValidationError[]}> = ({errors}) => { + const {t} = useTranslation() + return ( + + + {errors.map((error, index) => ( + + {translateSharedValidationError(t, error)} + + ))} + + + ) +} const CsvDiffView: React.FC<{previous: string; incoming: string}> = ({previous, incoming}) => { const diff = useMemo(() => diffLines(previous, incoming), [incoming, previous]) diff --git a/packages/admin-portal/src/translations/cat.ts b/packages/admin-portal/src/translations/cat.ts index 01d80331015..9313ccc1916 100644 --- a/packages/admin-portal/src/translations/cat.ts +++ b/packages/admin-portal/src/translations/cat.ts @@ -2212,6 +2212,10 @@ const catalanTranslation: TranslationType = { totalValidDoesNotMatch: "El total de vots vàlids no coincideix amb la suma dels vots dels candidats més els vots en blanc", censusTooSmall: "El cens ha de ser major o igual al total de vots", + totalInvalidDoesNotMatch: + "El total de vots invàlids ({{totalInvalid}}) ha de ser igual als vots invàlids implícits ({{implicitInvalid}}) més els vots invàlids explícits ({{explicitInvalid}})", + totalVotesDoesNotMatch: + "El total de vots ({{totalVotes}}) ha de ser igual al total de vots vàlids ({{totalValidVotes}}) més el total de vots invàlids ({{totalInvalid}})", }, label: { area: "Àrea", diff --git a/packages/admin-portal/src/translations/en.ts b/packages/admin-portal/src/translations/en.ts index db0ad6a031f..3baf4ba7b7a 100644 --- a/packages/admin-portal/src/translations/en.ts +++ b/packages/admin-portal/src/translations/en.ts @@ -2184,8 +2184,13 @@ const englishTranslation = { }, inputError: { totalValidDoesNotMatch: - "Total valid votes does not match the sum of the candidate votes plus blank votes", - censusTooSmall: "Census must be greater or equal than the total votes", + "Candidate votes ({{candidateVotesSum}}) must be between {{lowerBound}} and {{upperBound}} for this contest's voting rules ({{nonBlankValidVotes}} valid non-blank votes × up to {{maxMarks}} marks per ballot)", + censusTooSmall: + "Total votes ({{totalVotes}}) must not be greater than census ({{census}})", + totalInvalidDoesNotMatch: + "Total invalid votes ({{totalInvalid}}) must equal implicit invalid votes ({{implicitInvalid}}) plus explicit invalid votes ({{explicitInvalid}})", + totalVotesDoesNotMatch: + "Total votes ({{totalVotes}}) must equal total valid votes ({{totalValidVotes}}) plus total invalid votes ({{totalInvalid}})", }, label: { area: "Area", diff --git a/packages/admin-portal/src/translations/es.ts b/packages/admin-portal/src/translations/es.ts index 371e9ccc46b..06001b997f3 100644 --- a/packages/admin-portal/src/translations/es.ts +++ b/packages/admin-portal/src/translations/es.ts @@ -2205,6 +2205,10 @@ const spanishTranslation: TranslationType = { totalValidDoesNotMatch: "El total de votos válidos no coincide con la suma de los votos de los candidatos más los votos en blanco", censusTooSmall: "El censo debe ser mayor o igual al total de votos", + totalInvalidDoesNotMatch: + "El total de votos inválidos ({{totalInvalid}}) debe ser igual a los votos inválidos implícitos ({{implicitInvalid}}) más los votos inválidos explícitos ({{explicitInvalid}})", + totalVotesDoesNotMatch: + "El total de votos ({{totalVotes}}) debe ser igual al total de votos válidos ({{totalValidVotes}}) más el total de votos inválidos ({{totalInvalid}})", }, label: { area: "Area", diff --git a/packages/admin-portal/src/translations/eu.ts b/packages/admin-portal/src/translations/eu.ts index 1a74ab3f3bd..4a6559e22cd 100644 --- a/packages/admin-portal/src/translations/eu.ts +++ b/packages/admin-portal/src/translations/eu.ts @@ -2197,6 +2197,10 @@ const basqueTranslation: TranslationType = { "Baliozko botoen guztizko kopurua ez dator bat hautagaien botoen eta boto zurien baturarekin", censusTooSmall: "Erroldak boto guztien kopurua baino handiagoa edo berdina izan behar du", + totalInvalidDoesNotMatch: + "Boto baliogabeen guztizkoak ({{totalInvalid}}) boto baliogabe inplizituen ({{implicitInvalid}}) eta boto baliogabe esplizituen ({{explicitInvalid}}) baturaren berdina izan behar du", + totalVotesDoesNotMatch: + "Boto guztizkoak ({{totalVotes}}) boto baliodun guztizkoen ({{totalValidVotes}}) eta boto baliogabe guztizkoen ({{totalInvalid}}) baturaren berdina izan behar du", }, label: { area: "Eremua", diff --git a/packages/admin-portal/src/translations/fr.ts b/packages/admin-portal/src/translations/fr.ts index 60fbee35415..c02274d4ba8 100644 --- a/packages/admin-portal/src/translations/fr.ts +++ b/packages/admin-portal/src/translations/fr.ts @@ -2216,6 +2216,10 @@ const frenchTranslation: TranslationType = { totalValidDoesNotMatch: "Le total des votes valides ne correspond pas à la somme des votes des candidats plus les votes blancs", censusTooSmall: "Le recensement doit être supérieur ou égal au total des votes", + totalInvalidDoesNotMatch: + "Le total des votes invalides ({{totalInvalid}}) doit être égal aux votes invalides implicites ({{implicitInvalid}}) plus les votes invalides explicites ({{explicitInvalid}})", + totalVotesDoesNotMatch: + "Le total des votes ({{totalVotes}}) doit être égal au total des votes valides ({{totalValidVotes}}) plus le total des votes invalides ({{totalInvalid}})", }, label: { area: "Zone", diff --git a/packages/admin-portal/src/translations/gl.ts b/packages/admin-portal/src/translations/gl.ts index f1c4dbfc565..29abf4c19f3 100644 --- a/packages/admin-portal/src/translations/gl.ts +++ b/packages/admin-portal/src/translations/gl.ts @@ -2203,6 +2203,10 @@ const galegoTranslation: TranslationType = { totalValidDoesNotMatch: "O total de votos válidos non coincide coa suma dos votos dos candidatos máis os votos en branco", censusTooSmall: "O censo debe ser maior ou igual ao total de votos", + totalInvalidDoesNotMatch: + "O total de votos non válidos ({{totalInvalid}}) debe ser igual aos votos non válidos implícitos ({{implicitInvalid}}) máis os votos non válidos explícitos ({{explicitInvalid}})", + totalVotesDoesNotMatch: + "O total de votos ({{totalVotes}}) debe ser igual ao total de votos válidos ({{totalValidVotes}}) máis o total de votos non válidos ({{totalInvalid}})", }, label: { area: "Área", diff --git a/packages/admin-portal/src/translations/nl.ts b/packages/admin-portal/src/translations/nl.ts index 81a48dfee79..fad1fb7f50d 100644 --- a/packages/admin-portal/src/translations/nl.ts +++ b/packages/admin-portal/src/translations/nl.ts @@ -2199,6 +2199,10 @@ const dutchTranslation: TranslationType = { "Het totaal aantal geldige stemmen komt niet overeen met de som van de stemmen voor kandidaten plus blanco stemmen", censusTooSmall: "De census moet groter dan of gelijk zijn aan het totaal aantal stemmen", + totalInvalidDoesNotMatch: + "Het totaal aantal ongeldige stemmen ({{totalInvalid}}) moet gelijk zijn aan impliciet ongeldige stemmen ({{implicitInvalid}}) plus expliciet ongeldige stemmen ({{explicitInvalid}})", + totalVotesDoesNotMatch: + "Het totaal aantal stemmen ({{totalVotes}}) moet gelijk zijn aan het totaal aantal geldige stemmen ({{totalValidVotes}}) plus het totaal aantal ongeldige stemmen ({{totalInvalid}})", }, label: { area: "Gebied", diff --git a/packages/admin-portal/src/translations/tl.ts b/packages/admin-portal/src/translations/tl.ts index aafac90b22d..e2a3c8ee7f2 100644 --- a/packages/admin-portal/src/translations/tl.ts +++ b/packages/admin-portal/src/translations/tl.ts @@ -2209,6 +2209,10 @@ const tagalogTranslation: TranslationType = { "Ang kabuuang bilang ng mga balidong boto ay hindi tumutugma sa suma ng mga boto ng kandidato at mga blangkong boto", censusTooSmall: "Ang senso ay dapat na mas malaki o katumbas ng kabuuang bilang ng mga boto", + totalInvalidDoesNotMatch: + "Ang kabuuang bilang ng mga di-balidong boto ({{totalInvalid}}) ay dapat katumbas ng implicit na di-balidong boto ({{implicitInvalid}}) kasama ang explicit na di-balidong boto ({{explicitInvalid}})", + totalVotesDoesNotMatch: + "Ang kabuuang bilang ng boto ({{totalVotes}}) ay dapat katumbas ng kabuuang balidong boto ({{totalValidVotes}}) kasama ang kabuuang di-balidong boto ({{totalInvalid}})", }, label: { area: "Lugar", diff --git a/packages/ballot-verifier/rust/sequent-core-0.1.0.tgz b/packages/ballot-verifier/rust/sequent-core-0.1.0.tgz index 1912c3461cb..74a824cf49f 100644 Binary files a/packages/ballot-verifier/rust/sequent-core-0.1.0.tgz and b/packages/ballot-verifier/rust/sequent-core-0.1.0.tgz differ diff --git a/packages/harvest/src/routes/tally_sheets.rs b/packages/harvest/src/routes/tally_sheets.rs index 753df76c325..6a7fa3faea0 100644 --- a/packages/harvest/src/routes/tally_sheets.rs +++ b/packages/harvest/src/routes/tally_sheets.rs @@ -24,10 +24,10 @@ use sequent_core::types::tally_sheets::{ }; use serde::{Deserialize, Serialize}; use serde_json::Value; -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use tracing::{event, instrument, Level}; use windmill::postgres::{ - contest::get_contest_by_id, + contest::{export_contests, get_contest_by_id}, document::get_document, election_event::get_election_event_by_id, tally_session::{ @@ -41,7 +41,7 @@ use windmill::services::{ ceremonies::tally_ceremony::reset_tally_session_status_after_failed_recount_task, database::get_hasura_pool, documents::get_document_as_temp_file, - ess_xml_converter::convert_ess_enhanced_xml_to_csv, + ess_xml_converter::{convert_ess_enhanced_xml_to_csv, ContestVoteConfig}, tally_sheet_import::{ application::{ create_tally_sheet_import as create_tally_sheet_import_service, @@ -50,6 +50,7 @@ use windmill::services::{ }, errors::TallySheetImportError, hash::hash_bytes, + validation::contest_max_marks_per_ballot, }, }; use windmill::tasks::execute_tally_session::execute_tally_session; @@ -95,18 +96,6 @@ pub async fn create_new_tally_sheet( vec![Permissions::TALLY_SHEET_CREATE], )?; let input = body.into_inner(); - let validation_errors = validate_area_contest_results(&input.content); - if !validation_errors.is_empty() { - let messages = validation_errors - .into_iter() - .map(|error| format!("{}: {}", error.code, error.message)) - .collect::>() - .join("; "); - return Err(( - Status::BadRequest, - format!("Invalid tally sheet content: {messages}"), - )); - } let mut hasura_db_client: DbClient = get_hasura_pool() .await @@ -135,6 +124,22 @@ pub async fn create_new_tally_sheet( )); }; + let validation_errors = validate_area_contest_results( + &input.content, + contest_max_marks_per_ballot(&contest), + ); + if !validation_errors.is_empty() { + let messages = validation_errors + .into_iter() + .map(|error| format!("{}: {}", error.code, error.message)) + .collect::>() + .join("; "); + return Err(( + Status::BadRequest, + format!("Invalid tally sheet content: {messages}"), + )); + } + tally_sheet::lock_ballot_box_version_assignment( &hasura_transaction, &claims.hasura_claims.tenant_id, @@ -331,10 +336,18 @@ pub async fn preview_tally_sheet_import( .map_err(map_tally_sheet_import_error)?; verify_source_sha256(input.sha256.as_deref(), &source_bytes) .map_err(|e| (Status::BadRequest, format!("{e:?}")))?; + let contest_vote_config = contest_vote_config_by_external_id( + &hasura_transaction, + &claims.hasura_claims.tenant_id, + &input.election_event_id, + ) + .await + .map_err(|e| (Status::InternalServerError, format!("{:?}", e)))?; let (canonical_csv, conversion_validation_errors) = canonical_csv_bytes( &source_bytes, &input.source_format, &input.selected_channel, + &contest_vote_config, ) .map_err(|e| (Status::BadRequest, format!("{e:?}")))?; @@ -395,10 +408,18 @@ pub async fn create_tally_sheet_import( .map_err(map_tally_sheet_import_error)?; verify_source_sha256(input.sha256.as_deref(), &source_bytes) .map_err(|e| (Status::BadRequest, format!("{e:?}")))?; + let contest_vote_config = contest_vote_config_by_external_id( + &hasura_transaction, + &claims.hasura_claims.tenant_id, + &input.election_event_id, + ) + .await + .map_err(|e| (Status::InternalServerError, format!("{:?}", e)))?; let (canonical_csv, conversion_validation_errors) = canonical_csv_bytes( &source_bytes, &input.source_format, &input.selected_channel, + &contest_vote_config, ) .map_err(|e| (Status::BadRequest, format!("{e:?}")))?; @@ -733,6 +754,7 @@ fn canonical_csv_bytes( source_bytes: &[u8], source_format: &TallySheetImportSourceFormat, selected_channel: &VotingChannel, + contest_vote_config: &HashMap, ) -> Result<(Vec, Vec)> { match source_format { TallySheetImportSourceFormat::CANONICAL_CSV => { @@ -742,11 +764,42 @@ fn canonical_csv_bytes( convert_ess_enhanced_xml_to_csv( source_bytes, selected_channel.clone(), + contest_vote_config, ) } } } +/// Fetches every contest in the election event and maps its external id to +/// its `min_votes`/`max_votes`, for the ES&S converter to consult when +/// classifying under-votes and checking vote reconciliation (see +/// `convert_ess_enhanced_xml_to_csv`'s doc comment). Contests missing an +/// external id are skipped — the converter can't match ES&S rows to them +/// anyway. +async fn contest_vote_config_by_external_id( + hasura_transaction: &deadpool_postgres::Transaction<'_>, + tenant_id: &str, + election_event_id: &str, +) -> Result> { + let contests = + export_contests(hasura_transaction, tenant_id, election_event_id) + .await?; + Ok(contests + .into_iter() + .filter_map(|contest| { + contest.external_id.map(|external_id| { + ( + external_id, + ContestVoteConfig { + min_votes: contest.min_votes.unwrap_or(0), + max_votes: contest.max_votes.unwrap_or(1), + }, + ) + }) + }) + .collect()) +} + fn verify_source_sha256( expected_sha256: Option<&str>, source_bytes: &[u8], diff --git a/packages/sequent-core/src/services/tally_sheet_validation.rs b/packages/sequent-core/src/services/tally_sheet_validation.rs index a85725687a6..c58d0aa4046 100644 --- a/packages/sequent-core/src/services/tally_sheet_validation.rs +++ b/packages/sequent-core/src/services/tally_sheet_validation.rs @@ -4,16 +4,52 @@ use crate::types::tally_sheets::AreaContestResults; use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +/// `message` is a pre-formatted English sentence kept for logs, CLI output, +/// and any other non-translated context. `code` + `params` are what a UI +/// should use to render a translated message (`t(code, params)`); `params` +/// holds every number referenced by `message`, stringified. #[derive(Debug, Clone, Eq, PartialEq, Serialize, Deserialize)] pub struct TallySheetValidationError { pub code: String, pub message: String, pub field: String, + pub params: HashMap, +} + +/// Computes the maximum number of candidate marks a single non-blank +/// ballot can legitimately contribute, given a contest's `max_votes` and +/// counting algorithm. Absent or non-positive `max_votes` defaults to `1` +/// (single-choice), which reproduces the strict one-mark-per-ballot +/// behavior for ordinary contests. For `cumulative` contests, a voter may +/// give multiple points to the same candidate, so the bound is further +/// multiplied by the number of point checkboxes offered per candidate. +pub fn effective_max_marks_per_ballot( + max_votes: Option, + counting_algorithm: Option<&str>, + cumulative_number_of_checkboxes: Option, +) -> u64 { + let base = max_votes + .filter(|value| *value > 0) + .map(|value| value as u64) + .unwrap_or(1); + let is_cumulative = counting_algorithm + .map(|value| value.eq_ignore_ascii_case("cumulative")) + .unwrap_or(false); + if is_cumulative { + let checkboxes = cumulative_number_of_checkboxes + .filter(|value| *value > 0) + .unwrap_or(1); + base.saturating_mul(checkboxes) + } else { + base + } } pub fn validate_area_contest_results( content: &AreaContestResults, + max_marks_per_ballot: Option, ) -> Vec { let mut errors = Vec::new(); let invalid_votes = content.invalid_votes.clone().unwrap_or_default(); @@ -36,16 +72,39 @@ pub fn validate_area_contest_results( "total_invalid ({total_invalid}) must equal implicit_invalid ({implicit_invalid}) + explicit_invalid ({explicit_invalid})" ), "total_invalid", + HashMap::from([ + ("totalInvalid".to_string(), total_invalid.to_string()), + ("implicitInvalid".to_string(), implicit_invalid.to_string()), + ("explicitInvalid".to_string(), explicit_invalid.to_string()), + ]), )); } - if total_valid_votes != candidate_votes_sum + total_blank_votes { + // A voter may be allowed to mark more than one candidate per ballot + // (e.g. plurality-at-large "vote for N", or cumulative voting), so + // candidate_votes_sum isn't required to equal the ballot count — it + // must fall between one mark per non-blank ballot and + // max_marks_per_ballot marks per non-blank ballot. + let non_blank_valid_votes = + total_valid_votes.saturating_sub(total_blank_votes); + let max_marks = max_marks_per_ballot.unwrap_or(1).max(1); + let lower_bound = non_blank_valid_votes; + let upper_bound = non_blank_valid_votes.saturating_mul(max_marks); + + if candidate_votes_sum < lower_bound || candidate_votes_sum > upper_bound { errors.push(error( "invalid_total_valid_votes", format!( - "total_valid_votes ({total_valid_votes}) must equal candidate votes ({candidate_votes_sum}) + blank votes ({total_blank_votes})" + "candidate votes ({candidate_votes_sum}) must be between {lower_bound} and {upper_bound} (non-blank valid votes {non_blank_valid_votes} \u{d7} up to {max_marks} marks per ballot)" ), "total_valid_votes", + HashMap::from([ + ("candidateVotesSum".to_string(), candidate_votes_sum.to_string()), + ("lowerBound".to_string(), lower_bound.to_string()), + ("upperBound".to_string(), upper_bound.to_string()), + ("nonBlankValidVotes".to_string(), non_blank_valid_votes.to_string()), + ("maxMarks".to_string(), max_marks.to_string()), + ]), )); } @@ -56,6 +115,11 @@ pub fn validate_area_contest_results( "total_votes ({total_votes}) must equal total_valid_votes ({total_valid_votes}) + total_invalid ({total_invalid})" ), "total_votes", + HashMap::from([ + ("totalVotes".to_string(), total_votes.to_string()), + ("totalValidVotes".to_string(), total_valid_votes.to_string()), + ("totalInvalid".to_string(), total_invalid.to_string()), + ]), )); } @@ -69,6 +133,10 @@ pub fn validate_area_contest_results( "total_votes ({total_votes}) must not be greater than census ({census})" ), "census", + HashMap::from([ + ("totalVotes".to_string(), total_votes.to_string()), + ("census".to_string(), census.to_string()), + ]), )); } } @@ -80,11 +148,13 @@ fn error( code: &str, message: String, field: &str, + params: HashMap, ) -> TallySheetValidationError { TallySheetValidationError { code: code.to_string(), message, field: field.to_string(), + params, } } @@ -96,7 +166,9 @@ mod tests { AreaContestResults, CandidateResults, InvalidVotes, }; - use super::validate_area_contest_results; + use super::{ + effective_max_marks_per_ballot, validate_area_contest_results, + }; #[test] fn accepts_consistent_vote_bucket_arithmetic() { @@ -119,9 +191,10 @@ mod tests { total_votes: Some(10), }, )]), + annotations: None, }; - let errors = validate_area_contest_results(&content); + let errors = validate_area_contest_results(&content, None); assert!(errors.is_empty()); } @@ -147,9 +220,10 @@ mod tests { total_votes: Some(10), }, )]), + annotations: None, }; - let errors = validate_area_contest_results(&content); + let errors = validate_area_contest_results(&content, None); let codes = errors .into_iter() .map(|error| error.code) @@ -187,10 +261,205 @@ mod tests { total_votes: Some(10), }, )]), + annotations: None, + }; + + let errors = validate_area_contest_results(&content, None); + + assert!(errors.is_empty()); + } + + #[test] + fn accepts_vote_for_n_contest_within_bound() { + let content = AreaContestResults { + area_id: "area-1".to_string(), + contest_id: "contest-1".to_string(), + total_votes: Some(10), + total_valid_votes: Some(10), + invalid_votes: Some(InvalidVotes { + total_invalid: Some(0), + implicit_invalid: Some(0), + explicit_invalid: Some(0), + }), + total_blank_votes: Some(0), + census: Some(20), + candidate_results: HashMap::from([ + ( + "candidate-1".to_string(), + CandidateResults { + candidate_id: "candidate-1".to_string(), + total_votes: Some(8), + }, + ), + ( + "candidate-2".to_string(), + CandidateResults { + candidate_id: "candidate-2".to_string(), + total_votes: Some(7), + }, + ), + ]), + annotations: None, }; - let errors = validate_area_contest_results(&content); + let errors = validate_area_contest_results(&content, Some(2)); assert!(errors.is_empty()); } + + #[test] + fn rejects_vote_for_n_contest_exceeding_bound() { + let content = AreaContestResults { + area_id: "area-1".to_string(), + contest_id: "contest-1".to_string(), + total_votes: Some(10), + total_valid_votes: Some(10), + invalid_votes: Some(InvalidVotes { + total_invalid: Some(0), + implicit_invalid: Some(0), + explicit_invalid: Some(0), + }), + total_blank_votes: Some(0), + census: Some(30), + candidate_results: HashMap::from([ + ( + "candidate-1".to_string(), + CandidateResults { + candidate_id: "candidate-1".to_string(), + total_votes: Some(12), + }, + ), + ( + "candidate-2".to_string(), + CandidateResults { + candidate_id: "candidate-2".to_string(), + total_votes: Some(9), + }, + ), + ]), + annotations: None, + }; + + let errors = validate_area_contest_results(&content, Some(2)); + let codes = errors + .into_iter() + .map(|error| error.code) + .collect::>(); + + assert_eq!(codes, vec!["invalid_total_valid_votes"]); + } + + #[test] + fn rejects_vote_for_n_contest_below_lower_bound() { + let content = AreaContestResults { + area_id: "area-1".to_string(), + contest_id: "contest-1".to_string(), + total_votes: Some(10), + total_valid_votes: Some(10), + invalid_votes: Some(InvalidVotes { + total_invalid: Some(0), + implicit_invalid: Some(0), + explicit_invalid: Some(0), + }), + total_blank_votes: Some(0), + census: Some(20), + candidate_results: HashMap::from([( + "candidate-1".to_string(), + CandidateResults { + candidate_id: "candidate-1".to_string(), + total_votes: Some(5), + }, + )]), + annotations: None, + }; + + let errors = validate_area_contest_results(&content, Some(2)); + let codes = errors + .into_iter() + .map(|error| error.code) + .collect::>(); + + assert_eq!(codes, vec!["invalid_total_valid_votes"]); + } + + #[test] + fn accepts_cumulative_contest_using_full_checkbox_budget() { + let content = AreaContestResults { + area_id: "area-1".to_string(), + contest_id: "contest-1".to_string(), + total_votes: Some(10), + total_valid_votes: Some(10), + invalid_votes: Some(InvalidVotes { + total_invalid: Some(0), + implicit_invalid: Some(0), + explicit_invalid: Some(0), + }), + total_blank_votes: Some(0), + census: Some(60), + candidate_results: HashMap::from([( + "candidate-1".to_string(), + CandidateResults { + candidate_id: "candidate-1".to_string(), + total_votes: Some(40), + }, + )]), + annotations: None, + }; + + // max_votes=2, cumulative_number_of_checkboxes=3 -> 6 marks/ballot + let errors = validate_area_contest_results(&content, Some(6)); + + assert!(errors.is_empty()); + } + + #[test] + fn effective_max_marks_defaults_to_one_when_max_votes_absent() { + assert_eq!(effective_max_marks_per_ballot(None, None, None), 1); + } + + #[test] + fn effective_max_marks_floors_non_positive_max_votes_to_one() { + assert_eq!(effective_max_marks_per_ballot(Some(0), None, None), 1); + assert_eq!(effective_max_marks_per_ballot(Some(-1), None, None), 1); + } + + #[test] + fn effective_max_marks_uses_max_votes_for_non_cumulative_contests() { + assert_eq!( + effective_max_marks_per_ballot( + Some(3), + Some("plurality-at-large"), + None + ), + 3 + ); + } + + #[test] + fn effective_max_marks_multiplies_by_checkboxes_for_cumulative_contests() { + assert_eq!( + effective_max_marks_per_ballot( + Some(2), + Some("cumulative"), + Some(3) + ), + 6 + ); + assert_eq!( + effective_max_marks_per_ballot( + Some(2), + Some("Cumulative"), + Some(3) + ), + 6 + ); + } + + #[test] + fn effective_max_marks_defaults_checkboxes_to_one_when_absent() { + assert_eq!( + effective_max_marks_per_ballot(Some(2), Some("cumulative"), None), + 2 + ); + } } diff --git a/packages/sequent-core/src/types/tally_sheet_import.rs b/packages/sequent-core/src/types/tally_sheet_import.rs index 35c491e824f..5b5be88190c 100644 --- a/packages/sequent-core/src/types/tally_sheet_import.rs +++ b/packages/sequent-core/src/types/tally_sheet_import.rs @@ -4,6 +4,8 @@ #![allow(non_camel_case_types)] +use std::collections::HashMap; + use serde::{Deserialize, Serialize}; use serde_json::Value; use strum_macros::{Display, EnumString}; @@ -56,6 +58,11 @@ pub enum TallySheetImportReviewDecision { DISAPPROVE, } +/// `message` is a pre-formatted English sentence kept for logs, CLI output, +/// and any other non-translated context. `code` + `params` are what a UI +/// should use to render a translated message (`t(code, params)`); `params` +/// is empty for structural/parsing errors that don't reduce to a fixed set +/// of translation keys (e.g. malformed CSV rows or XML). #[derive(Debug, Clone, Eq, PartialEq, Serialize, Deserialize)] pub struct TallySheetImportValidationError { pub code: String, @@ -65,6 +72,7 @@ pub struct TallySheetImportValidationError { pub contest_external_id: Option, pub candidate_external_id: Option, pub field: Option, + pub params: HashMap, } #[derive(Debug, Clone, Eq, PartialEq, Serialize, Deserialize, Default)] diff --git a/packages/sequent-core/src/types/tally_sheets.rs b/packages/sequent-core/src/types/tally_sheets.rs index c50fe624b41..3fdcb62cd4f 100644 --- a/packages/sequent-core/src/types/tally_sheets.rs +++ b/packages/sequent-core/src/types/tally_sheets.rs @@ -4,6 +4,7 @@ #![allow(non_camel_case_types)] use serde::{Deserialize, Serialize}; +use serde_json::Value; use std::collections::HashMap; use std::str::FromStr; use strum_macros::{Display, EnumString}; @@ -79,4 +80,8 @@ pub struct AreaContestResults { pub total_blank_votes: Option, pub census: Option, pub candidate_results: HashMap, + /// Free-form extra data from the source tally system that doesn't have + /// a dedicated field + #[serde(default)] + pub annotations: Option, } diff --git a/packages/sequent-core/src/wasm/areas.rs b/packages/sequent-core/src/wasm/areas.rs index 27cec6ad74f..3115d2664e9 100644 --- a/packages/sequent-core/src/wasm/areas.rs +++ b/packages/sequent-core/src/wasm/areas.rs @@ -3,14 +3,16 @@ // SPDX-License-Identifier: AGPL-3.0-only use crate::services::area_tree::*; -use crate::services::tally_sheet_validation::validate_area_contest_results; +use crate::services::tally_sheet_validation::{ + effective_max_marks_per_ballot, validate_area_contest_results, +}; use crate::types::hasura::core::AreaContest; use crate::types::tally_sheets::AreaContestResults; use crate::wasm::wasm::IntoResult; use std::collections::HashSet; use wasm_bindgen::prelude::*; extern crate console_error_panic_hook; -use serde::Serialize; +use serde::{Deserialize, Serialize}; use serde_wasm_bindgen; use serde_wasm_bindgen::Serializer; use std::panic; @@ -68,10 +70,19 @@ pub fn get_contest_matches_js( .into_json() } +#[derive(Deserialize)] +#[serde(rename_all = "snake_case")] +struct ContestMarkBoundsInput { + max_votes: Option, + counting_algorithm: Option, + cumulative_number_of_checkboxes: Option, +} + #[allow(clippy::all)] #[wasm_bindgen] pub fn validate_area_contest_results_js( content_json: JsValue, + contest_bounds_json: JsValue, ) -> Result { let content: AreaContestResults = serde_wasm_bindgen::from_value(content_json).map_err(|err| { @@ -80,8 +91,22 @@ pub fn validate_area_contest_results_js( err ) })?; + let bounds: ContestMarkBoundsInput = + serde_wasm_bindgen::from_value(contest_bounds_json).map_err(|err| { + format!( + "Error reading javascript contest bounds for validation: {}", + err + ) + })?; + + let max_marks_per_ballot = effective_max_marks_per_ballot( + bounds.max_votes, + bounds.counting_algorithm.as_deref(), + bounds.cumulative_number_of_checkboxes, + ); - let errors = validate_area_contest_results(&content); + let errors = + validate_area_contest_results(&content, Some(max_marks_per_ballot)); let serializer = Serializer::json_compatible(); errors .serialize(&serializer) diff --git a/packages/step-cli/src/commands/tally_sheet.rs b/packages/step-cli/src/commands/tally_sheet.rs index 953cc6c3f7e..6a0805c7d0c 100644 --- a/packages/step-cli/src/commands/tally_sheet.rs +++ b/packages/step-cli/src/commands/tally_sheet.rs @@ -18,6 +18,7 @@ use serde::{de::DeserializeOwned, Serialize}; use serde_json::Value; use sha2::{Digest, Sha256}; use std::{ + collections::HashMap, error::Error, fs::{self, File}, io::Read, @@ -712,8 +713,12 @@ pub fn convert_ess_xml_to_tally_csv( selected_channel: VotingChannelArg, ) -> Result<(), Box> { let xml_bytes = fs::read(input)?; + // This is a standalone, offline conversion (no election event context), + // so contest min_votes can't be looked up — every contest is treated as + // min_votes=0, meaning under-votes never count as invalid. That's the + // common case; see convert_ess_enhanced_xml_to_csv's doc comment. let (csv_bytes, validation_errors) = - convert_ess_enhanced_xml_to_csv(&xml_bytes, selected_channel.to_core())?; + convert_ess_enhanced_xml_to_csv(&xml_bytes, selected_channel.to_core(), &HashMap::new())?; fs::write(output, csv_bytes)?; for error in &validation_errors { let context = match &error.contest_external_id { diff --git a/packages/ui-core/rust/sequent-core-0.1.0.tgz b/packages/ui-core/rust/sequent-core-0.1.0.tgz index 1912c3461cb..74a824cf49f 100644 Binary files a/packages/ui-core/rust/sequent-core-0.1.0.tgz and b/packages/ui-core/rust/sequent-core-0.1.0.tgz differ diff --git a/packages/velvet/src/pipes/do_tally/tally.rs b/packages/velvet/src/pipes/do_tally/tally.rs index faf59959989..147f7c92833 100644 --- a/packages/velvet/src/pipes/do_tally/tally.rs +++ b/packages/velvet/src/pipes/do_tally/tally.rs @@ -439,6 +439,7 @@ mod tests { total_blank_votes: Some(blank_votes), census: Some(candidate_votes + blank_votes + 1), candidate_results, + annotations: None, }), channel: None, deleted_at: None, diff --git a/packages/voting-portal/rust/sequent-core-0.1.0.tgz b/packages/voting-portal/rust/sequent-core-0.1.0.tgz index 1912c3461cb..74a824cf49f 100644 Binary files a/packages/voting-portal/rust/sequent-core-0.1.0.tgz and b/packages/voting-portal/rust/sequent-core-0.1.0.tgz differ diff --git a/packages/windmill/src/services/ess_xml_converter.rs b/packages/windmill/src/services/ess_xml_converter.rs index e018ee9cacd..30c228c024a 100644 --- a/packages/windmill/src/services/ess_xml_converter.rs +++ b/packages/windmill/src/services/ess_xml_converter.rs @@ -16,9 +16,32 @@ use tracing::instrument; /// read by default when a caller doesn't ask for a specific one. pub const DEFAULT_IMPORT_REPORTING_GROUP_ID: &str = "1"; +/// The STEP contest config this converter needs — `min_votes` decides +/// whether undervoting can invalidate a ballot; `max_votes` is the "vote +/// for N" bound used to check ES&S's reported totals reconcile exactly. +/// Missing/non-positive values default conservatively (`min_votes` to `0`, +/// `max_votes` to `1`, i.e. single-choice). +#[derive(Debug, Clone, Copy, Default)] +pub struct ContestVoteConfig { + pub min_votes: i64, + pub max_votes: i64, +} + #[derive(Debug, Clone, Default)] struct ContestPrecinctTotals { - ballots_cast: u64, + /// Ballots cast across the whole precinct (every contest/ballot style + /// on it) — always available, used only as the `census` upper bound. + precinct_ballots_cast: u64, + /// Ballots cast specifically for *this* contest in this precinct, when + /// the XML variant reports it at that granularity (the + /// `ContestReportingGroupVotes` variant does). `None` for the + /// candidate-reporting-group variant: its `ballotsCast` lives on the + /// `` element, shared across every contest on the ballot, so + /// it isn't a valid `total_votes` for a contest that only appears on + /// some ballot styles within that precinct (e.g. a ward- or + /// school-board-specific race) — see the doc comment on + /// `convert_ess_enhanced_xml_to_csv` for how each case is handled. + contest_ballots_cast: Option, over_votes: u64, under_votes: u64, blank_votes: u64, @@ -47,9 +70,88 @@ fn xml_error( contest_external_id: contest_external_id.map(|id| id.to_string()), candidate_external_id: None, field: None, + params: HashMap::new(), } } +/// Checks the exact per-ballot accounting identity documented in the ES&S +/// EVS SOP for `overVotes`/`underVotes`: every one of a contest's +/// `max_votes` selection slots, across every ballot cast for it, ends up +/// as either a candidate mark, an overvote slot, or an undervote slot, +/// with no remainder — so `candidate_votes_sum + over_votes + under_votes` +/// must equal `total_votes * max_votes` exactly. Returns a validation +/// error scoped to this precinct/contest when it doesn't (a data-quality +/// problem in the source file, not something a caller can fix). +#[allow(clippy::too_many_arguments)] +fn check_vote_reconciliation( + selected_channel: &VotingChannel, + area_name: &str, + contest_external_id: &str, + candidate_votes_sum: u64, + over_votes: u64, + under_votes: u64, + total_votes: u64, + max_votes: u64, +) -> Option { + let expected = total_votes.saturating_mul(max_votes); + let actual = candidate_votes_sum + over_votes + under_votes; + if actual == expected { + return None; + } + Some(TallySheetImportValidationError { + code: "ess_vote_reconciliation_mismatch".to_string(), + message: format!( + "candidate votes ({candidate_votes_sum}) + over votes ({over_votes}) + under votes ({under_votes}) must equal total votes ({total_votes}) \u{d7} {max_votes} marks per ballot ({expected})" + ), + channel: Some(selected_channel.clone()), + area_name: Some(area_name.to_string()), + contest_external_id: Some(contest_external_id.to_string()), + candidate_external_id: None, + field: Some("total_valid_votes".to_string()), + params: HashMap::from([ + ("candidateVotesSum".to_string(), candidate_votes_sum.to_string()), + ("overVotes".to_string(), over_votes.to_string()), + ("underVotes".to_string(), under_votes.to_string()), + ("totalVotes".to_string(), total_votes.to_string()), + ("maxVotes".to_string(), max_votes.to_string()), + ("expected".to_string(), expected.to_string()), + ]), + }) +} + +/// Checks that `overVotes` is an exact multiple of `max_votes`, for the +/// `ContestReportingGroupVotes` variant. ES&S always attributes a whole +/// `max_votes` allotment to `overVotes` for every overvoted ballot +/// (confirmed by the EVS SOP and empirically, no partial-slot overvotes +/// observed) — so a non-exact remainder indicates a data-quality problem in +/// the source file, not something this importer can resolve on its own. +fn check_over_votes_divisible( + selected_channel: &VotingChannel, + area_name: &str, + contest_external_id: &str, + over_votes: u64, + max_votes: u64, +) -> Option { + if over_votes % max_votes == 0 { + return None; + } + Some(TallySheetImportValidationError { + code: "ess_over_votes_not_divisible".to_string(), + message: format!( + "over votes ({over_votes}) is not an exact multiple of {max_votes} marks per ballot — cannot determine the number of overvoted ballots" + ), + channel: Some(selected_channel.clone()), + area_name: Some(area_name.to_string()), + contest_external_id: Some(contest_external_id.to_string()), + candidate_external_id: None, + field: Some("implicit_invalid".to_string()), + params: HashMap::from([ + ("overVotes".to_string(), over_votes.to_string()), + ("maxVotes".to_string(), max_votes.to_string()), + ]), + }) +} + /// Resolves a single contest's precinct totals and candidate votes, /// regardless of which of the two ES&S XML variants it uses. fn resolve_contest_data( @@ -74,11 +176,13 @@ fn resolve_contest_data( pub fn convert_ess_enhanced_xml_to_csv( xml_bytes: &[u8], selected_channel: VotingChannel, + contest_vote_config: &HashMap, ) -> Result<(Vec, Vec)> { convert_ess_enhanced_xml_to_csv_for_reporting_group( xml_bytes, selected_channel, DEFAULT_IMPORT_REPORTING_GROUP_ID, + contest_vote_config, ) } @@ -92,11 +196,31 @@ pub fn convert_ess_enhanced_xml_to_csv( /// `TallySheetImportValidationError`, exactly like `parse_canonical_csv` /// does for a bad CSV row — so a file with one broken contest still /// converts and imports every other contest in it. +/// +/// `contest_vote_config` maps a contest's external id to its `min_votes`/ +/// `max_votes` (STEP contest config); a contest missing from the map uses +/// the defaults documented on `ContestVoteConfig`. `min_votes` decides +/// whether undervoting can invalidate a ballot — see the comment above +/// `implicit_invalid` below. `max_votes` is used to check ES&S's reported +/// totals reconcile exactly — see the comment above `check_vote_reconciliation`. +/// +/// `total_votes`/`total_valid_votes` are derived differently depending on +/// the XML variant. The `ContestReportingGroupVotes` variant reports +/// `ballotsCast` per contest *and* precinct, so it's used directly. The +/// candidate-reporting-group variant only reports `ballotsCast` per +/// precinct (shared across every contest on the ballot), which isn't a +/// valid ballot count for a contest that doesn't appear on every ballot +/// style in that precinct (e.g. a ward- or school-board-specific race) — +/// for that variant, `total_valid_votes` is derived from candidate marks +/// plus blank votes instead, the only figures actually scoped to this +/// contest and precinct. `census` always uses the precinct-wide ballots +/// cast, regardless of variant. #[instrument(skip_all, err)] pub fn convert_ess_enhanced_xml_to_csv_for_reporting_group( xml_bytes: &[u8], selected_channel: VotingChannel, reporting_group_id: &str, + contest_vote_config: &HashMap, ) -> Result<(Vec, Vec)> { let xml = std::str::from_utf8(xml_bytes).context("ES&S XML import must be valid UTF-8")?; let document = Document::parse(xml).context("Invalid ES&S Enhanced XML")?; @@ -142,6 +266,13 @@ pub fn convert_ess_enhanced_xml_to_csv_for_reporting_group( } }; + let vote_config = contest_vote_config + .get(&contest_external_id) + .copied() + .unwrap_or_default(); + let min_votes = vote_config.min_votes; + let max_votes = vote_config.max_votes.max(1) as u64; + for (precinct_id, totals) in totals_by_precinct { let Some(area_name) = precinct_names.get(&precinct_id) else { validation_errors.push(xml_error( @@ -155,6 +286,60 @@ pub fn convert_ess_enhanced_xml_to_csv_for_reporting_group( continue; }; + // For the ContestReportingGroupVotes variant (contest_ballots_cast + // is Some), ES&S's own `blankVotes` is not a genuine blank-ballot + // count — it's exactly `overVotes + underVotes` (confirmed both + // by the EVS SOP's field description and empirically, with no + // exceptions). `underVotes` alone is the figure that actually + // reconciles as a ballot-cardinality "blank" for single-choice + // contests: `ballots_cast == candidate_votes_sum + underVotes` + // exactly whenever max_votes == 1. The candidate-reporting-group + // variant doesn't have this problem — its blank figure comes + // from the precinct's own `blanksCast`, a genuine (if + // precinct-wide) blank-ballot count — so it keeps using that. + let total_blank_votes = if totals.contest_ballots_cast.is_some() { + totals.under_votes + } else { + totals.blank_votes + }; + // Overvoting a "vote for N" contest always spoils that contest + // on the ballot, regardless of over_vote_policy — see + // sequent_core::ballot_codec::checker::check_over_vote_policy, + // whose policy branches only change alert UI copy; the + // invalid_errors push (the actual invalidity decision) is + // unconditional there. + // + // ES&S's overVotes is a *selection-slot* count, not a ballot + // count (see check_over_votes_divisible's doc comment). For the + // ContestReportingGroupVotes variant, total_votes below is a + // genuine ballot count from ES&S, so overVotes must be divided + // by max_votes to recover an overvoted-*ballot* count — + // otherwise implicit_invalid could exceed total_votes and make + // total_valid_votes underflow to 0. The other variant derives + // total_votes from candidate marks/blank votes instead (see the + // total_votes/total_valid_votes match below), so it isn't + // exposed to that underflow and keeps using the raw slot count. + let mut implicit_invalid = if totals.contest_ballots_cast.is_some() { + totals.over_votes / max_votes + } else { + totals.over_votes + }; + // Undervoting only invalidates a ballot when it selects fewer + // candidates than min_votes requires — see + // check_under_vote_policy's comment: falling short of + // min_votes "is an invalid vote no matter what" the + // under_vote_policy is; under_vote_policy itself never + // invalidates a ballot on its own. ES&S's aggregate under_votes + // count doesn't distinguish "fell short of min_votes" from any + // other undervote, so it can't be attributed precisely — this + // is a best-effort approximation (and a no-op whenever + // total_blank_votes is already under_votes itself, i.e. the + // ContestReportingGroupVotes variant above). + if min_votes > 0 { + implicit_invalid += totals.under_votes.saturating_sub(total_blank_votes); + } + let explicit_invalid = 0; + let total_invalid = implicit_invalid + explicit_invalid; let candidate_votes_sum = candidates .iter() .map(|candidate| { @@ -165,12 +350,28 @@ pub fn convert_ess_enhanced_xml_to_csv_for_reporting_group( .unwrap_or(0) }) .sum::(); - let total_blank_votes = totals.blank_votes; - let implicit_invalid = - totals.over_votes + totals.under_votes.saturating_sub(total_blank_votes); - let explicit_invalid = 0; - let total_valid_votes = candidate_votes_sum + total_blank_votes; - let total_votes = total_valid_votes + implicit_invalid + explicit_invalid; + let (total_votes, total_valid_votes) = match totals.contest_ballots_cast { + // A single ballot can legitimately carry more than one + // candidate mark (e.g. "vote for N" contests), so + // total_votes comes from ES&S's own per-contest + // ballots-cast figure rather than being derived from the + // sum of candidate marks. + Some(contest_ballots_cast) => { + let total_votes = contest_ballots_cast; + let total_valid_votes = total_votes.saturating_sub(total_invalid); + (total_votes, total_valid_votes) + } + // No ballots-cast figure is scoped to this contest and + // precinct (candidate-reporting-group variant) — the only + // figures that are scoped that way are the candidate marks + // and blank votes, so total_valid_votes is derived from + // those instead, same as for a single-choice contest. + None => { + let total_valid_votes = candidate_votes_sum + total_blank_votes; + let total_votes = total_valid_votes + total_invalid; + (total_votes, total_valid_votes) + } + }; write_scalar_row( &mut writer, @@ -218,9 +419,53 @@ pub fn convert_ess_enhanced_xml_to_csv_for_reporting_group( area_name, &contest_external_id, "census", - totals.ballots_cast, + totals.precinct_ballots_cast, )?; + // Only meaningful when total_votes is ES&S's own per-contest + // ballots-cast figure (not derived from candidate marks) — see + // check_vote_reconciliation's doc comment for the identity + // being checked. + if totals.contest_ballots_cast.is_some() { + write_scalar_row( + &mut writer, + &selected_channel, + area_name, + &contest_external_id, + "over_votes", + totals.over_votes, + )?; + write_scalar_row( + &mut writer, + &selected_channel, + area_name, + &contest_external_id, + "under_votes", + totals.under_votes, + )?; + if let Some(error) = check_over_votes_divisible( + &selected_channel, + area_name, + &contest_external_id, + totals.over_votes, + max_votes, + ) { + validation_errors.push(error); + } + if let Some(error) = check_vote_reconciliation( + &selected_channel, + area_name, + &contest_external_id, + candidate_votes_sum, + totals.over_votes, + totals.under_votes, + total_votes, + max_votes, + ) { + validation_errors.push(error); + } + } + for candidate in &candidates { writer.write_record([ selected_channel.to_string(), @@ -280,8 +525,10 @@ fn contest_totals_by_precinct( { let precinct_id = required_attr(votes, "refPrecinctId", "ContestReportingGroupVotes")?; let entry = totals_by_precinct.entry(precinct_id).or_default(); - entry.ballots_cast += - parse_u64_attr(votes, "ballotsCast", "ContestReportingGroupVotes")?; + let ballots_cast = parse_u64_attr(votes, "ballotsCast", "ContestReportingGroupVotes")?; + entry.precinct_ballots_cast += ballots_cast; + entry.contest_ballots_cast = + Some(entry.contest_ballots_cast.unwrap_or(0) + ballots_cast); entry.over_votes += parse_u64_attr(votes, "overVotes", "ContestReportingGroupVotes")?; entry.under_votes += parse_u64_attr(votes, "underVotes", "ContestReportingGroupVotes")?; entry.blank_votes += parse_u64_attr(votes, "blankVotes", "ContestReportingGroupVotes")?; @@ -434,7 +681,7 @@ fn precinct_reporting_group_totals_by_precinct( found_import_group = true; let entry = totals_by_precinct.entry(precinct_id.clone()).or_default(); - entry.ballots_cast += + entry.precinct_ballots_cast += parse_u64_attr(reporting_group, "ballotsCast", "PrecinctReportingGroup")?; entry.blank_votes += parse_u64_attr(reporting_group, "blanksCast", "PrecinctReportingGroup")?; @@ -538,7 +785,7 @@ mod tests { - + @@ -550,7 +797,79 @@ mod tests { "#; - let (csv, errors) = convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER).unwrap(); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()).unwrap(); + let csv = String::from_utf8(csv).unwrap(); + + assert!(errors.is_empty()); + // min_votes defaults to 0 for a contest missing from the map, so + // the 3 under-votes here never invalidate a ballot — only the 2 + // over-votes do. candidate_sum(13) + over(2) + under(5) == + // ballots_cast(20) * max_votes(1, the default), satisfying the + // reconciliation check. + assert!(csv.contains("PAPER,Precinct 1,contest-1,implicit_invalid,,2")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_blank_votes,,5")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_valid_votes,,18")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_votes,,20")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,candidate_votes,cand-1,10")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,over_votes,,2")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,under_votes,,5")); + assert!(!csv.contains("ignored-overvotes")); + } + + #[test] + fn undervotes_only_count_as_invalid_when_contest_requires_a_minimum() { + // The candidate-reporting-group variant, where total_blank_votes + // (from the precinct's own blanksCast) and under_votes (from this + // contest's UNDERVOTES pseudo-candidate) are independent figures, + // so the overlap-safe subtraction has an observable effect. This + // contest requires at least 1 selection (min_votes=1), so the + // portion of under_votes beyond the precinct's blank ballots is + // folded into implicit_invalid. + let xml = br#" + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + "#; + + let contest_vote_config = HashMap::from([( + "contest-1".to_string(), + ContestVoteConfig { + min_votes: 1, + max_votes: 1, + }, + )]); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &contest_vote_config) + .unwrap(); let csv = String::from_utf8(csv).unwrap(); assert!(errors.is_empty()); @@ -558,8 +877,57 @@ mod tests { assert!(csv.contains("PAPER,Precinct 1,contest-1,total_blank_votes,,4")); assert!(csv.contains("PAPER,Precinct 1,contest-1,total_valid_votes,,14")); assert!(csv.contains("PAPER,Precinct 1,contest-1,total_votes,,17")); - assert!(csv.contains("PAPER,Precinct 1,contest-1,candidate_votes,cand-1,7")); - assert!(!csv.contains("ignored-overvotes")); + // candidate-reporting-group variant: no per-contest ballots-cast, + // so no over_votes/under_votes annotations are emitted either. + assert!(!csv.contains("contest-1,over_votes")); + assert!(!csv.contains("contest-1,under_votes")); + } + + #[test] + fn derives_totals_from_ballots_cast_not_candidate_marks() { + // A "vote for 2" contest: candidate marks (15 + 12 = 27) legitimately + // exceed ballots_cast (20) because each ballot can select up to 2 + // candidates. total_votes/total_valid_votes must still be derived + // from ballots_cast, not from summing candidate marks. underVotes + // (13) accounts for the rest of the 40 available slots (20 * 2) + // not used by a candidate mark, satisfying the reconciliation + // check: 27 + 0 + 13 == 20 * 2. + let xml = br#" + + + + + + + + + + + + + + + + + "#; + + let contest_vote_config = HashMap::from([( + "contest-1".to_string(), + ContestVoteConfig { + min_votes: 0, + max_votes: 2, + }, + )]); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &contest_vote_config) + .unwrap(); + let csv = String::from_utf8(csv).unwrap(); + + assert!(errors.is_empty()); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_votes,,20")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_valid_votes,,20")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,candidate_votes,cand-1,15")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,candidate_votes,cand-2,12")); } #[test] @@ -602,7 +970,8 @@ mod tests { "#; - let (csv, errors) = convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER).unwrap(); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()).unwrap(); let csv = String::from_utf8(csv).unwrap(); assert!(errors.is_empty()); @@ -615,6 +984,45 @@ mod tests { assert!(csv.contains("PAPER,Precinct 1,contest-1,total_votes,,100")); } + #[test] + fn derives_totals_from_marks_for_a_contest_not_on_every_ballot_style() { + // The candidate-reporting-group variant's ballotsCast is reported + // per precinct only (shared across every contest on the ballot), + // not per contest. A contest that only appears on some ballot + // styles in the precinct (e.g. a school-board trustee race) will + // have far fewer candidate marks than the precinct's ballotsCast — + // total_votes must come from those marks, not the precinct total. + let xml = br#" + + + + + + + + + + + + + + + + + "#; + + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()).unwrap(); + let csv = String::from_utf8(csv).unwrap(); + + assert!(errors.is_empty()); + assert!(csv.contains("PAPER,Precinct 1,contest-1,candidate_votes,cand-1,5")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_blank_votes,,1")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_valid_votes,,6")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_votes,,6")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,census,,100")); + } + #[test] fn ignores_group_zero_when_group_one_exists() { let xml = br#" @@ -640,7 +1048,8 @@ mod tests { "#; - let (csv, errors) = convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER).unwrap(); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()).unwrap(); let csv = String::from_utf8(csv).unwrap(); assert!(errors.is_empty()); @@ -664,33 +1073,41 @@ mod tests { - - + + "#; + // Ward 1: candidate_sum(17) + over(1) + under(2) == ballots_cast(20) * max_votes(1). + // Ward 2: candidate_sum(10) + over(0) + under(0) == ballots_cast(10) * max_votes(1). let expected = "\ channel,area_name,contest_external_id,field,candidate_external_id,value -PAPER,Ward 1,contest-1,total_votes,,10 -PAPER,Ward 1,contest-1,total_valid_votes,,9 +PAPER,Ward 1,contest-1,total_votes,,20 +PAPER,Ward 1,contest-1,total_valid_votes,,19 PAPER,Ward 1,contest-1,implicit_invalid,,1 PAPER,Ward 1,contest-1,explicit_invalid,,0 PAPER,Ward 1,contest-1,total_blank_votes,,2 PAPER,Ward 1,contest-1,census,,20 -PAPER,Ward 1,contest-1,candidate_votes,cand-1,7 -PAPER,Ward 2,contest-1,total_votes,,5 -PAPER,Ward 2,contest-1,total_valid_votes,,5 +PAPER,Ward 1,contest-1,over_votes,,1 +PAPER,Ward 1,contest-1,under_votes,,2 +PAPER,Ward 1,contest-1,candidate_votes,cand-1,17 +PAPER,Ward 2,contest-1,total_votes,,10 +PAPER,Ward 2,contest-1,total_valid_votes,,10 PAPER,Ward 2,contest-1,implicit_invalid,,0 PAPER,Ward 2,contest-1,explicit_invalid,,0 -PAPER,Ward 2,contest-1,total_blank_votes,,1 +PAPER,Ward 2,contest-1,total_blank_votes,,0 PAPER,Ward 2,contest-1,census,,10 -PAPER,Ward 2,contest-1,candidate_votes,cand-1,4 +PAPER,Ward 2,contest-1,over_votes,,0 +PAPER,Ward 2,contest-1,under_votes,,0 +PAPER,Ward 2,contest-1,candidate_votes,cand-1,10 "; for _ in 0..5 { - let (csv, errors) = convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER).unwrap(); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()) + .unwrap(); assert!(errors.is_empty()); assert_eq!(String::from_utf8(csv).unwrap(), expected); } @@ -720,7 +1137,8 @@ PAPER,Ward 2,contest-1,candidate_votes,cand-1,4 // A structural problem scoped to one contest is a validation error, // not a hard failure: the conversion still succeeds (with an empty // canonical CSV, since this file's only contest is the broken one). - let (csv, errors) = convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER).unwrap(); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()).unwrap(); assert_eq!(errors.len(), 1); assert_eq!(errors[0].code, "xml_conversion_error"); assert_eq!(errors[0].contest_external_id, Some("contest-1".to_string())); @@ -754,7 +1172,7 @@ PAPER,Ward 2,contest-1,candidate_votes,cand-1,4 - + @@ -763,7 +1181,8 @@ PAPER,Ward 2,contest-1,candidate_votes,cand-1,4 "#; - let (csv, errors) = convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER).unwrap(); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()).unwrap(); let csv = String::from_utf8(csv).unwrap(); assert_eq!(errors.len(), 1); @@ -776,8 +1195,139 @@ PAPER,Ward 2,contest-1,candidate_votes,cand-1,4 .contains("Duplicate CandidatePrecinctVotes")); assert!(!csv.contains("broken-contest")); - assert!(csv.contains("PAPER,Precinct 1,good-contest,candidate_votes,cand-1,7")); + assert!(csv.contains("PAPER,Precinct 1,good-contest,candidate_votes,cand-1,10")); assert!(csv.contains("PAPER,Precinct 1,good-contest,candidate_votes,cand-2,3")); - assert!(csv.contains("PAPER,Precinct 1,good-contest,total_votes,,17")); + assert!(csv.contains("PAPER,Precinct 1,good-contest,total_votes,,20")); + } + + #[test] + fn reports_a_validation_error_when_totals_do_not_reconcile() { + // candidate_sum(5) + over(2) + under(5) = 12, but ballots_cast(20) + // * max_votes(1, the default) = 20 — a genuine data-quality + // problem in the source file, per the ES&S SOP's per-ballot + // accounting identity. + let xml = br#" + + + + + + + + + + + + + + "#; + + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &HashMap::new()).unwrap(); + let csv = String::from_utf8(csv).unwrap(); + + assert_eq!(errors.len(), 1); + assert_eq!(errors[0].code, "ess_vote_reconciliation_mismatch"); + assert_eq!(errors[0].contest_external_id, Some("contest-1".to_string())); + assert_eq!(errors[0].area_name, Some("Precinct 1".to_string())); + // The rest of the contest still converts — this is a validation + // warning about the source data, not a hard failure. + assert!(csv.contains("PAPER,Precinct 1,contest-1,candidate_votes,cand-1,5")); + } + + #[test] + fn overvoted_ballot_on_a_vote_for_n_contest_counts_as_one_invalid_ballot() { + // Reproduces a real ES&S test file: 3 ballots on a voteFor=4 + // contest — one voted correctly for cand-1..cand-4, one fully + // overvoted, one left entirely blank. The overvoted ballot + // contributes its whole max_votes allotment (4) to overVotes, and + // the blank ballot contributes its whole allotment to underVotes. + // implicit_invalid must come from overVotes / max_votes (1 + // overvoted *ballot*), not the raw overVotes slot count (4) — + // otherwise total_valid_votes (3 ballots - 4 invalid) would + // underflow to 0 instead of the correct 2. + let xml = br#" + + + + + + + + + + + + + + + + + + + + + + + "#; + + let contest_vote_config = HashMap::from([( + "contest-1".to_string(), + ContestVoteConfig { + min_votes: 0, + max_votes: 4, + }, + )]); + let (csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &contest_vote_config) + .unwrap(); + let csv = String::from_utf8(csv).unwrap(); + + assert!(errors.is_empty()); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_votes,,3")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,implicit_invalid,,1")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,total_valid_votes,,2")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,over_votes,,4")); + assert!(csv.contains("PAPER,Precinct 1,contest-1,under_votes,,4")); + } + + #[test] + fn reports_a_validation_error_when_over_votes_is_not_an_exact_multiple_of_max_votes() { + // candidate_sum(3) + over(5) + under(4) = 12 = ballots_cast(3) * + // max_votes(4), so the reconciliation identity holds — but overVotes + // (5) isn't itself an exact multiple of max_votes (4), so there's no + // way to know how many ballots it represents. This is a distinct + // data-quality problem from a reconciliation mismatch. + let xml = br#" + + + + + + + + + + + + + + "#; + + let contest_vote_config = HashMap::from([( + "contest-1".to_string(), + ContestVoteConfig { + min_votes: 0, + max_votes: 4, + }, + )]); + let (_csv, errors) = + convert_ess_enhanced_xml_to_csv(xml, VotingChannel::PAPER, &contest_vote_config) + .unwrap(); + + assert_eq!(errors.len(), 1); + assert_eq!(errors[0].code, "ess_over_votes_not_divisible"); + assert_eq!(errors[0].contest_external_id, Some("contest-1".to_string())); + assert_eq!(errors[0].area_name, Some("Precinct 1".to_string())); } } diff --git a/packages/windmill/src/services/tally_sheet_import/application.rs b/packages/windmill/src/services/tally_sheet_import/application.rs index 668ec342066..1d7079f486a 100644 --- a/packages/windmill/src/services/tally_sheet_import/application.rs +++ b/packages/windmill/src/services/tally_sheet_import/application.rs @@ -45,7 +45,7 @@ use super::{ diff::{classify_change, render_ballot_box_csv}, errors::TallySheetImportError, hash::{hash_area_contest_results, hash_bytes}, - validation::validate_import_content, + validation::{contest_max_marks_per_ballot, validate_import_content}, }; const DISPLAY_NAME_FALLBACK_LANG: &str = "en"; @@ -110,6 +110,13 @@ pub async fn preview_tally_sheet_import( contest_external_id: Some(parsed_import.key.contest_external_id.clone()), candidate_external_id: None, field: Some("channel".to_string()), + params: HashMap::from([ + ( + "rowChannel".to_string(), + parsed_import.key.channel.to_string(), + ), + ("selectedChannel".to_string(), selected_channel.to_string()), + ]), }); continue; } @@ -135,6 +142,7 @@ pub async fn preview_tally_sheet_import( contest_external_id: Some(parsed_import.key.contest_external_id.clone()), candidate_external_id: None, field: None, + params: HashMap::new(), }); continue; } @@ -145,6 +153,7 @@ pub async fn preview_tally_sheet_import( &parsed_import.key.area_name, &parsed_import.key.contest_external_id, &resolved.content, + contest_max_marks_per_ballot(&resolved.contest), )); let ballot_box_key = BallotBoxKey { diff --git a/packages/windmill/src/services/tally_sheet_import/csv.rs b/packages/windmill/src/services/tally_sheet_import/csv.rs index e4b719df67f..f0e75e71436 100644 --- a/packages/windmill/src/services/tally_sheet_import/csv.rs +++ b/packages/windmill/src/services/tally_sheet_import/csv.rs @@ -11,6 +11,7 @@ use sequent_core::types::tally_sheets::{ AreaContestResults, CandidateResults, InvalidVotes, VotingChannel, }; use serde::Deserialize; +use serde_json::Value; use tracing::instrument; #[derive(Debug, Clone, Eq, PartialEq, Hash)] @@ -26,7 +27,7 @@ pub struct ParsedBallotBoxImport { pub content: AreaContestResults, } -#[derive(Debug, Clone, Copy, Eq, PartialEq)] +#[derive(Debug, Clone, Eq, PartialEq)] enum CanonicalField { CandidateVotes, TotalBlankVotes, @@ -35,6 +36,12 @@ enum CanonicalField { TotalValidVotes, TotalVotes, Census, + /// Any field name outside the fixed set above. Carried through as an + /// unvalidated `annotations` entry (no duplicate-row or + /// required-field checks) so source-specific extra data (e.g. ES&S's + /// `over_votes`/`under_votes`) — or fields added in the future — don't + /// need a parser change to flow through. + Annotation(String), } impl FromStr for CanonicalField { @@ -49,7 +56,7 @@ impl FromStr for CanonicalField { "total_valid_votes" => Ok(Self::TotalValidVotes), "total_votes" => Ok(Self::TotalVotes), "census" => Ok(Self::Census), - _ => Err(()), + other => Ok(Self::Annotation(other.to_string())), } } } @@ -73,6 +80,10 @@ struct BallotBoxAccumulator { total_blank_votes: Option, census: Option, candidate_results: HashMap, + /// Values from any field row outside the fixed scalar set, keyed by + /// field name. Last value wins for a repeated key — unlike the fixed + /// scalar fields, duplicates aren't treated as an error here. + annotations: HashMap, } #[instrument(skip_all)] @@ -198,6 +209,17 @@ pub fn parse_canonical_csv( implicit_invalid: accumulator.implicit_invalid, explicit_invalid: accumulator.explicit_invalid, }; + let annotations = if accumulator.annotations.is_empty() { + None + } else { + Some(Value::Object( + accumulator + .annotations + .into_iter() + .map(|(name, value)| (name, Value::from(value))) + .collect(), + )) + }; imports.push(ParsedBallotBoxImport { key, content: AreaContestResults { @@ -209,6 +231,7 @@ pub fn parse_canonical_csv( total_blank_votes: accumulator.total_blank_votes, census: accumulator.census, candidate_results: accumulator.candidate_results, + annotations, }, }); } @@ -303,6 +326,9 @@ fn apply_row( CanonicalField::Census => { set_scalar(validation_errors, key, row, &mut accumulator.census, value) } + CanonicalField::Annotation(name) => { + accumulator.annotations.insert(name, value); + } } } @@ -370,6 +396,7 @@ fn error_for_row( contest_external_id, candidate_external_id, field, + params: HashMap::new(), } } diff --git a/packages/windmill/src/services/tally_sheet_import/hash.rs b/packages/windmill/src/services/tally_sheet_import/hash.rs index 0c9d2af03c8..9a30254f9bf 100644 --- a/packages/windmill/src/services/tally_sheet_import/hash.rs +++ b/packages/windmill/src/services/tally_sheet_import/hash.rs @@ -135,6 +135,7 @@ mod tests { total_blank_votes: Some(1), census: Some(25), candidate_results, + annotations: None, } } } diff --git a/packages/windmill/src/services/tally_sheet_import/validation.rs b/packages/windmill/src/services/tally_sheet_import/validation.rs index e634e98b5bd..6667a7456c1 100644 --- a/packages/windmill/src/services/tally_sheet_import/validation.rs +++ b/packages/windmill/src/services/tally_sheet_import/validation.rs @@ -2,19 +2,42 @@ // // SPDX-License-Identifier: AGPL-3.0-only -use sequent_core::services::tally_sheet_validation::validate_area_contest_results; +use std::collections::HashMap; + +use sequent_core::services::tally_sheet_validation::{ + effective_max_marks_per_ballot, validate_area_contest_results, +}; +use sequent_core::types::hasura::core::Contest; use sequent_core::types::tally_sheet_import::TallySheetImportValidationError; use sequent_core::types::tally_sheets::{AreaContestResults, VotingChannel}; use tracing::instrument; +/// Computes the maximum number of candidate marks a single non-blank +/// ballot can legitimately contribute for the given contest, from its +/// `max_votes`, `counting_algorithm`, and (for cumulative contests) the +/// per-candidate checkbox budget stored in `presentation`. +pub fn contest_max_marks_per_ballot(contest: &Contest) -> Option { + let cumulative_number_of_checkboxes = contest + .presentation + .as_ref() + .and_then(|value| value.get("cumulative_number_of_checkboxes")) + .and_then(|value| value.as_u64()); + Some(effective_max_marks_per_ballot( + contest.max_votes, + contest.counting_algorithm.as_deref(), + cumulative_number_of_checkboxes, + )) +} + #[instrument(skip_all)] pub fn validate_import_content( channel: &VotingChannel, area_name: &str, contest_external_id: &str, content: &AreaContestResults, + max_marks_per_ballot: Option, ) -> Vec { - validate_area_contest_results(content) + validate_area_contest_results(content, max_marks_per_ballot) .into_iter() .map(|shared_error| { error( @@ -24,6 +47,7 @@ pub fn validate_import_content( area_name, contest_external_id, &shared_error.field, + shared_error.params, ) }) .collect() @@ -36,6 +60,7 @@ fn error( area_name: &str, contest_external_id: &str, field: &str, + params: HashMap, ) -> TallySheetImportValidationError { TallySheetImportValidationError { code: code.to_string(), @@ -45,6 +70,7 @@ fn error( contest_external_id: Some(contest_external_id.to_string()), candidate_external_id: None, field: Some(field.to_string()), + params, } } @@ -77,10 +103,16 @@ mod tests { total_votes: Some(10), }, )]), + annotations: None, }; - let errors = - validate_import_content(&VotingChannel::PAPER, "Precinct 1", "contest-1", &content); + let errors = validate_import_content( + &VotingChannel::PAPER, + "Precinct 1", + "contest-1", + &content, + None, + ); assert!(errors.is_empty()); } @@ -106,10 +138,16 @@ mod tests { total_votes: Some(10), }, )]), + annotations: None, }; - let errors = - validate_import_content(&VotingChannel::PAPER, "Precinct 1", "contest-1", &content); + let errors = validate_import_content( + &VotingChannel::PAPER, + "Precinct 1", + "contest-1", + &content, + None, + ); let codes = errors .into_iter() .map(|error| error.code) @@ -125,4 +163,96 @@ mod tests { ] ); } + + #[test] + fn accepts_vote_for_n_contest_within_bound() { + let content = AreaContestResults { + area_id: "area-1".to_string(), + contest_id: "contest-1".to_string(), + total_votes: Some(10), + total_valid_votes: Some(10), + invalid_votes: Some(InvalidVotes { + total_invalid: Some(0), + implicit_invalid: Some(0), + explicit_invalid: Some(0), + }), + total_blank_votes: Some(0), + census: Some(20), + candidate_results: HashMap::from([ + ( + "candidate-1".to_string(), + CandidateResults { + candidate_id: "candidate-1".to_string(), + total_votes: Some(8), + }, + ), + ( + "candidate-2".to_string(), + CandidateResults { + candidate_id: "candidate-2".to_string(), + total_votes: Some(7), + }, + ), + ]), + annotations: None, + }; + + let errors = validate_import_content( + &VotingChannel::PAPER, + "Precinct 1", + "contest-1", + &content, + Some(2), + ); + + assert!(errors.is_empty()); + } + + #[test] + fn rejects_vote_for_n_contest_exceeding_bound() { + let content = AreaContestResults { + area_id: "area-1".to_string(), + contest_id: "contest-1".to_string(), + total_votes: Some(10), + total_valid_votes: Some(10), + invalid_votes: Some(InvalidVotes { + total_invalid: Some(0), + implicit_invalid: Some(0), + explicit_invalid: Some(0), + }), + total_blank_votes: Some(0), + census: Some(30), + candidate_results: HashMap::from([ + ( + "candidate-1".to_string(), + CandidateResults { + candidate_id: "candidate-1".to_string(), + total_votes: Some(12), + }, + ), + ( + "candidate-2".to_string(), + CandidateResults { + candidate_id: "candidate-2".to_string(), + total_votes: Some(9), + }, + ), + ]), + annotations: None, + }; + + let errors = validate_import_content( + &VotingChannel::PAPER, + "Precinct 1", + "contest-1", + &content, + Some(2), + ); + let codes = errors + .into_iter() + .map(|error| error.code) + .collect::>(); + + assert_eq!(codes, vec!["invalid_total_valid_votes"]); + } } diff --git a/packages/yarn.lock b/packages/yarn.lock index e909f8e2cc5..7ec78034558 100644 --- a/packages/yarn.lock +++ b/packages/yarn.lock @@ -17502,15 +17502,15 @@ sentence-case@^3.0.4: "sequent-core@file:./admin-portal/rust/sequent-core-0.1.0.tgz": version "0.1.0" - resolved "file:./admin-portal/rust/sequent-core-0.1.0.tgz#72e87d0083e6a2a12d1c5494f64ef79a3f5d6221" + resolved "file:./admin-portal/rust/sequent-core-0.1.0.tgz#b37c67ff69d34dda4e7402375bfda5bb9b20c40d" "sequent-core@file:./ballot-verifier/rust/sequent-core-0.1.0.tgz": version "0.1.0" - resolved "file:./ballot-verifier/rust/sequent-core-0.1.0.tgz#72e87d0083e6a2a12d1c5494f64ef79a3f5d6221" + resolved "file:./ballot-verifier/rust/sequent-core-0.1.0.tgz#b37c67ff69d34dda4e7402375bfda5bb9b20c40d" "sequent-core@file:./voting-portal/rust/sequent-core-0.1.0.tgz": version "0.1.0" - resolved "file:./voting-portal/rust/sequent-core-0.1.0.tgz#72e87d0083e6a2a12d1c5494f64ef79a3f5d6221" + resolved "file:./voting-portal/rust/sequent-core-0.1.0.tgz#b37c67ff69d34dda4e7402375bfda5bb9b20c40d" serialize-javascript@^6.0.0, serialize-javascript@^6.0.2: version "6.0.2"