🐞 Tally Sheet ES&S import - "Vote For N" contest values for blank and invalid and valid votes are incorrect

docs/docusaurus/docs/02-election_managers/02-reference/02-election-event/08-03-election_management_election-event_tally-sheet-imports.md

@@ -46,7 +46,7 @@ channel + area_name + contest_external_id
 
 The selected import channel must match every `channel` value in the CSV.
 
-Each ballot box must include exactly one row for every scalar field: `total_votes`, `total_valid_votes`, `implicit_invalid`, `explicit_invalid`, `total_blank_votes`, and `census`. It must also include one `candidate_votes` row for every candidate in the matched STEP contest.
+Each ballot box must include exactly one row for every required scalar field: `total_votes`, `total_valid_votes`, `implicit_invalid`, `explicit_invalid`, `total_blank_votes`, and `census`. It must also include one `candidate_votes` row for every candidate in the matched STEP contest. Any other `field` value (e.g. `over_votes`/`under_votes`, written by the ES&S import path — see below) is accepted without validation and carried through as extra data on the imported ballot box, so new source-specific fields don't need a canonical CSV format change to be picked up.
 
 Uploaded files are stored as normal STEP documents. The UI and CLI send a SHA-256 hash of the source file when they upload it, and the import actions verify that hash before parsing or persisting the import. The import record stores both the original source hash and the canonical CSV hash used for validation and review.
 
@@ -58,7 +58,7 @@ Uploaded files are stored as normal STEP documents. The UI and CLI send a SHA-25
 
 `contest_external_id` must match the contest external ID configured in STEP. Because the canonical CSV format has no election column, **contest external IDs must be unique within an election event** for tally sheet import to resolve rows correctly. If an election event has multiple elections that reuse the same external ID for different contests, the import will fail with an ambiguous-match error — rename the duplicate external IDs before importing.
 
-`field` must be one of:
+`field` is usually one of the required scalar fields or `candidate_votes`:
 
 - `candidate_votes`
 - `total_blank_votes`
@@ -68,6 +68,8 @@ Uploaded files are stored as normal STEP documents. The UI and CLI send a SHA-25
 - `total_votes`
 - `census`
 
+Any other value (e.g. `over_votes`, `under_votes`) is accepted as unvalidated extra data — see the note above.
+
 `candidate_external_id` is required only when `field` is `candidate_votes`. It must match a candidate external ID in the matched contest.
 
 `value` must be a non-negative integer.
@@ -86,11 +88,14 @@ The importer validates these invariants per ballot box:
 
 ```text
 total_invalid = implicit_invalid + explicit_invalid
-total_valid_votes = sum(candidate_votes) + total_blank_votes
+non_blank_valid_votes = total_valid_votes - total_blank_votes
+non_blank_valid_votes <= sum(candidate_votes) <= non_blank_valid_votes * max_marks_per_ballot
 total_votes = total_valid_votes + total_invalid
 total_votes <= census
 ```
 
+A voter may be allowed to mark more than one candidate per ballot, so `sum(candidate_votes)` isn't required to equal the ballot count — it only has to fall within the range a valid ballot for this contest could produce. `max_marks_per_ballot` is `1` for ordinary single-choice contests, the contest's `max_votes` for plurality-at-large "vote for N" contests, and `max_votes` multiplied by the cumulative-voting checkbox limit for cumulative contests.
+
 ## ES&S Enhanced XML Mapping
 
 For ES&S Enhanced XML uploads:
@@ -99,9 +104,12 @@ For ES&S Enhanced XML uploads:
 - contest `altId1` is matched to contest external ID
 - candidate `altId1` is matched to candidate external ID
 - only ES&S reporting group `1` is read: its values are summed per precinct into one result for the selected STEP channel, and all other reporting groups are ignored
-- ES&S overvotes become STEP implicit invalid votes
-- ES&S blank votes become STEP blank votes
-- ES&S undervotes are treated with the overlap-safe rule `max(underVotes - blankVotes, 0)` before adding them to implicit invalid votes
+- ES&S overvotes always become STEP implicit invalid votes. ES&S's `overVotes` is a selection-*slot* count, not a ballot count — an overvoted ballot always contributes its whole `max_votes` allotment to `overVotes` (confirmed by the EVS SOP and empirically). For contests where ballots-cast is reported per contest and precinct (the `ContestReportingGroupVotes` XML variant), `overVotes` is divided by `max_votes` to recover an overvoted-*ballot* count before it's added to `implicit_invalid`; otherwise it could exceed `total_votes` and make `total_valid_votes` underflow. The other ES&S variant derives `total_votes` from candidate marks/blank votes instead (see below), so it isn't exposed to that underflow and uses the raw `overVotes` count directly.
+- ES&S undervotes only become implicit invalid votes (via the overlap-safe rule `max(underVotes - blankVotes, 0)`) when the contest requires a minimum number of selections (`min_votes > 0`) — undervoting an otherwise-optional contest is never invalid
+- ES&S's own `blankVotes` figure is **not** used as STEP's `total_blank_votes` for contests where ballots-cast is reported per contest and precinct (the `ContestReportingGroupVotes` XML variant): per the ES&S EVS documentation, `blankVotes` is exactly `overVotes + underVotes`, not a genuine blank-ballot count, so it can't validate a "vote for N" contest correctly. `underVotes` alone is used instead, which reconciles exactly for single-choice contests. The other ES&S variant (candidate-reporting-group) doesn't have this problem — its blank figure comes from the precinct's own `blanksCast`, a genuine (if precinct-wide) blank-ballot count — so it keeps using that.
+- `total_votes`/`total_valid_votes` are derived from ES&S's own ballots-cast figure when it's reported per contest *and* precinct (the `ContestReportingGroupVotes` XML variant), so vote-for-N contests (where candidate marks legitimately exceed the ballot count) convert correctly. The other ES&S variant only reports ballots cast per precinct, shared across every contest on the ballot — not a valid ballot count for a contest that doesn't appear on every ballot style in that precinct (e.g. a ward- or school-board-specific race). For that variant, `total_valid_votes` is derived from candidate marks plus blank votes instead, the only figures actually scoped to that contest and precinct; `census` always uses the precinct-wide ballots cast regardless of variant.
+- for the `ContestReportingGroupVotes` variant, the import additionally writes `over_votes`/`under_votes` rows carrying ES&S's raw counts, and cross-checks the per-ballot accounting identity documented in the ES&S SOP: every one of a contest's `max_votes` selection slots, across every ballot cast for it, is either a candidate mark, an overvote slot, or an undervote slot, with no remainder (`sum(candidate_votes) + over_votes + under_votes == total_votes * max_votes`). A mismatch is reported as an `ess_vote_reconciliation_mismatch` validation error — it indicates a data-quality problem in the source file, not something STEP can correct on its own.
+- for the `ContestReportingGroupVotes` variant, the import also checks that `over_votes` is an exact multiple of `max_votes` — required for the overvoted-ballot count used to compute `implicit_invalid` (see above) to be well-defined. A non-exact remainder is reported as an `ess_over_votes_not_divisible` validation error.
 
 ## Import Lifecycle
 

packages/admin-portal/src/resources/TallySheet/EditTallySheet.tsx

@@ -31,7 +31,7 @@ import {
 } from "@mui/material"
 import TextField from "@mui/material/TextField"
 import {IAreaContestResults, ICandidateResults, IInvalidVotes} from "@/types/TallySheets"
-import {sortFunction} from "./utils"
+import {sortFunction, translateSharedValidationError} from "./utils"
 import {
     EEnableCheckableLists,
     ICandidatePresentation,
@@ -86,10 +86,19 @@ interface SharedValidationError {
     code: string
     message: string
     field: string
+    params: Record<string, string>
 }
 
-const validateAreaContestResults = (content: IAreaContestResults): SharedValidationError[] =>
-    validate_area_contest_results_js(content)
+interface IContestMarkBounds {
+    max_votes?: Maybe<number>
+    counting_algorithm?: Maybe<string>
+    cumulative_number_of_checkboxes?: number
+}
+
+const validateAreaContestResults = (
+    content: IAreaContestResults,
+    contestBounds: IContestMarkBounds
+): SharedValidationError[] => validate_area_contest_results_js(content, contestBounds)
 
 const numbersRegExp = /^[0-9]+$/
 
@@ -107,6 +116,9 @@ export const EditTallySheet: React.FC<EditTallySheetProps> = (props) => {
     const {t, i18n} = useTranslation()
     const aliasRenderer = useAliasRenderer()
 
+    const translateValidationError = (error: SharedValidationError): string =>
+        translateSharedValidationError(t, error)
+
     const [areasList, setAreasList] = useState<IArea[]>([])
     const [contestList, setContestList] = useState<IContest[]>([])
     const [channel, setChannel] = React.useState<string | null>(null)
@@ -125,9 +137,9 @@ export const EditTallySheet: React.FC<EditTallySheetProps> = (props) => {
     const [areaNameFilter, setAreaNameFilter] = useState<string | null>(null)
     const [contestNameFilter, setContestNameFilter] = useState<string | null>(null)
     const [areaIds, setAreaIds] = useState<Array<string>>([])
-    const [totalValidError, setTotalValidError] = useState<boolean>(false)
-    const [censusError, setCensusError] = useState<boolean>(false)
-    const [sharedValidationMessages, setSharedValidationMessages] = useState<string[]>([])
+    const [sharedValidationErrors, setSharedValidationErrors] = useState<SharedValidationError[]>(
+        []
+    )
     const {data: areaContests} = useGetList<Sequent_Backend_Area_Contest>(
         "sequent_backend_area_contest",
         {
@@ -375,24 +387,22 @@ export const EditTallySheet: React.FC<EditTallySheetProps> = (props) => {
             }
         }
 
-        const sharedValidationErrors = validateAreaContestResults({
-            ...newResults,
-            invalid_votes: invalids,
-            candidate_results: candidateResultsForValidation,
-        })
-
-        const codes = new Set(sharedValidationErrors.map((error) => error.code))
-        setTotalValidError(codes.has("invalid_total_valid_votes"))
-        setCensusError(codes.has("total_votes_exceeds_census"))
-        setSharedValidationMessages(
-            sharedValidationErrors
-                .filter(
-                    (error) =>
-                        error.code !== "invalid_total_valid_votes" &&
-                        error.code !== "total_votes_exceeds_census"
-                )
-                .map((error) => error.message)
+        const contestPresentation = choosenContest?.presentation as IContestPresentation | undefined
+        const sharedValidationErrors = validateAreaContestResults(
+            {
+                ...newResults,
+                invalid_votes: invalids,
+                candidate_results: candidateResultsForValidation,
+            },
+            {
+                max_votes: choosenContest?.max_votes,
+                counting_algorithm: choosenContest?.counting_algorithm,
+                cumulative_number_of_checkboxes:
+                    contestPresentation?.cumulative_number_of_checkboxes,
+            }
         )
+
+        setSharedValidationErrors(sharedValidationErrors)
         setIsButtonDisabled(sharedValidationErrors.length > 0)
 
         if (JSON.stringify(newResults) !== JSON.stringify(results)) {
@@ -407,6 +417,7 @@ export const EditTallySheet: React.FC<EditTallySheetProps> = (props) => {
         results.total_valid_votes,
         invalids?.total_invalid,
         invalids,
+        choosenContest,
     ])
 
     const handleChange = (
@@ -588,6 +599,18 @@ export const EditTallySheet: React.FC<EditTallySheetProps> = (props) => {
         }
     }, [choosenContest])
 
+    const totalValidVotesError = sharedValidationErrors.find(
+        (error) => error.code === "invalid_total_valid_votes"
+    )
+    const censusExceededError = sharedValidationErrors.find(
+        (error) => error.code === "total_votes_exceeds_census"
+    )
+    const otherValidationErrors = sharedValidationErrors.filter(
+        (error) =>
+            error.code !== "invalid_total_valid_votes" &&
+            error.code !== "total_votes_exceeds_census"
+    )
+
     return (
         <SimpleForm toolbar={false} onSubmit={onSubmit}>
             <>
@@ -693,13 +716,13 @@ export const EditTallySheet: React.FC<EditTallySheetProps> = (props) => {
                         size="small"
                         required
                     />
-                    {totalValidError && (
-                        <StyledError>
-                            {t("tallysheet.inputError.totalValidDoesNotMatch")}
-                        </StyledError>
+                    {totalValidVotesError && (
+                        <StyledError>{translateValidationError(totalValidVotesError)}</StyledError>
                     )}
-                    {sharedValidationMessages.map((message) => (
-                        <StyledError key={message}>{message}</StyledError>
+                    {otherValidationErrors.map((error) => (
+                        <StyledError key={error.code + error.field}>
+                            {translateValidationError(error)}
+                        </StyledError>
                     ))}
                 </>
                 <Box
@@ -770,8 +793,8 @@ export const EditTallySheet: React.FC<EditTallySheetProps> = (props) => {
                         size="small"
                         required
                     />
-                    {censusError && (
-                        <StyledError>{t("tallysheet.inputError.censusTooSmall")}</StyledError>
+                    {censusExceededError && (
+                        <StyledError>{translateValidationError(censusExceededError)}</StyledError>
                     )}
                 </>
                 <PageHeaderStyles.Wrapper>

packages/admin-portal/src/resources/TallySheet/utils/index.ts

@@ -3,10 +3,46 @@
 // SPDX-License-Identifier: AGPL-3.0-only
 
 import {isString} from "@sequentech/ui-core"
+import {useTranslation} from "react-i18next"
 
 export const sortFunction = (a: {name?: string | null}, b: {name?: string | null}) => {
     if (isString(a?.name) && isString(b?.name)) {
         return a.name.localeCompare(b.name)
     }
     return 0
 }
+
+export interface ISharedValidationError {
+    code: string
+    message: string
+    field?: string | null
+    params?: Record<string, string> | null
+}
+
+// Maps a shared tally sheet validation error's `code` (from
+// sequent_core::services::tally_sheet_validation, used by both manual entry
+// and tally sheet import) to the i18n key used to render it. Codes without
+// an entry here fall back to `tallysheet.inputError.<code>`.
+const VALIDATION_ERROR_TRANSLATION_KEYS: Record<string, string> = {
+    invalid_total_valid_votes: "tallysheet.inputError.totalValidDoesNotMatch",
+    total_votes_exceeds_census: "tallysheet.inputError.censusTooSmall",
+    invalid_total_invalid: "tallysheet.inputError.totalInvalidDoesNotMatch",
+    invalid_total_votes: "tallysheet.inputError.totalVotesDoesNotMatch",
+}
+
+// `t` is typed loosely (matching react-i18next's `useTranslation().t`)
+// rather than importing i18next's TFunction, to avoid a new type
+// dependency for a single helper.
+export const translateSharedValidationError = (
+    t: ReturnType<typeof useTranslation>["t"],
+    error: ISharedValidationError
+): string => {
+    const translationKey =
+        VALIDATION_ERROR_TRANSLATION_KEYS[error.code] ?? `tallysheet.inputError.${error.code}`
+    return String(
+        t(translationKey, {
+            ...(error.params ?? {}),
+            defaultValue: error.message,
+        })
+    )
+}

packages/admin-portal/src/resources/TallySheetImport/TallySheetImports.tsx

@@ -48,6 +48,7 @@ import {
     useRefresh,
 } from "react-admin"
 import {useTranslation} from "react-i18next"
+import {translateSharedValidationError} from "@/resources/TallySheet/utils"
 import ElectionHeader from "@/components/ElectionHeader"
 import {ListActions} from "@/components/ListActions"
 import {GET_UPLOAD_URL} from "@/queries/GetUploadUrl"
@@ -101,6 +102,7 @@ interface TallySheetImportValidationError {
     contest_external_id?: string | null
     candidate_external_id?: string | null
     field?: string | null
+    params?: Record<string, string> | null
 }
 
 interface TallySheetImportPreviewItem {
@@ -1229,17 +1231,20 @@ const SummaryChip: React.FC<{label: string; value: number}> = ({label, value}) =
     </Box>
 )
 
-const ValidationErrors: React.FC<{errors: TallySheetImportValidationError[]}> = ({errors}) => (
-    <Alert severity="error">
-        <Stack gap={0.5}>
-            {errors.map((error, index) => (
-                <Typography key={`${error.code}:${index}`} variant="body2">
-                    {error.message}
-                </Typography>
-            ))}
-        </Stack>
-    </Alert>
-)
+const ValidationErrors: React.FC<{errors: TallySheetImportValidationError[]}> = ({errors}) => {
+    const {t} = useTranslation()
+    return (
+        <Alert severity="error">
+            <Stack gap={0.5}>
+                {errors.map((error, index) => (
+                    <Typography key={`${error.code}:${index}`} variant="body2">
+                        {translateSharedValidationError(t, error)}
+                    </Typography>
+                ))}
+            </Stack>
+        </Alert>
+    )
+}
 
 const CsvDiffView: React.FC<{previous: string; incoming: string}> = ({previous, incoming}) => {
     const diff = useMemo(() => diffLines(previous, incoming), [incoming, previous])

packages/admin-portal/src/translations/cat.ts

@@ -2212,6 +2212,10 @@ const catalanTranslation: TranslationType = {
                 totalValidDoesNotMatch:
                     "El total de vots vàlids no coincideix amb la suma dels vots dels candidats més els vots en blanc",
                 censusTooSmall: "El cens ha de ser major o igual al total de vots",
+                totalInvalidDoesNotMatch:
+                    "El total de vots invàlids ({{totalInvalid}}) ha de ser igual als vots invàlids implícits ({{implicitInvalid}}) més els vots invàlids explícits ({{explicitInvalid}})",
+                totalVotesDoesNotMatch:
+                    "El total de vots ({{totalVotes}}) ha de ser igual al total de vots vàlids ({{totalValidVotes}}) més el total de vots invàlids ({{totalInvalid}})",
             },
             label: {
                 area: "Àrea",

packages/admin-portal/src/translations/en.ts

@@ -2184,8 +2184,13 @@ const englishTranslation = {
             },
             inputError: {
                 totalValidDoesNotMatch:
-                    "Total valid votes does not match the sum of the candidate votes plus blank votes",
-                censusTooSmall: "Census must be greater or equal than the total votes",
+                    "Candidate votes ({{candidateVotesSum}}) must be between {{lowerBound}} and {{upperBound}} for this contest's voting rules ({{nonBlankValidVotes}} valid non-blank votes × up to {{maxMarks}} marks per ballot)",
+                censusTooSmall:
+                    "Total votes ({{totalVotes}}) must not be greater than census ({{census}})",
+                totalInvalidDoesNotMatch:
+                    "Total invalid votes ({{totalInvalid}}) must equal implicit invalid votes ({{implicitInvalid}}) plus explicit invalid votes ({{explicitInvalid}})",
+                totalVotesDoesNotMatch:
+                    "Total votes ({{totalVotes}}) must equal total valid votes ({{totalValidVotes}}) plus total invalid votes ({{totalInvalid}})",
             },
             label: {
                 area: "Area",