Wave 2: datascope CSV mixed-date P1 fix + 2.4.0 bump

.github/workflows/publish.yml

@@ -15,7 +15,12 @@ jobs:
       id-token: write
     steps:
       - uses: actions/checkout@v5
-      - uses: actions/setup-python@v5
+
+      - name: Engagement guard
+        run: python3 scripts/engagement_guard.py
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
         with:
           python-version: "3.12"
       - run: python -m pip install build

.gitignore

@@ -50,3 +50,11 @@ secrets.*
 
 # UI-review tool output (regenerated each run)
 screenshots/
+
+# --- lailara engagement scaffold ---
+# Client engagement data is runtime-only: never commit it, never deploy it.
+client-data/
+client-output/
+/engagement.yml
+/engagement.yaml
+# (engagement.demo.yml and engagement.example.yml stay committable)

CHANGELOG.md

@@ -2,6 +2,23 @@
 
 All notable changes to datascope are documented here.
 
+## [Unreleased]
+
+### Fixed
+- HTML report generator timestamp now honors `SOURCE_DATE_EPOCH` (the reproducible-builds standard), so regenerated sample reports are byte-identical run-to-run. A bare `datetime.now()` in the report footer otherwise changed every run and defeated any byte-lock on the output.
+
+### Changed
+- Regenerated `samples/output/` from current source with `SOURCE_DATE_EPOCH` pinned, so the shipped showcase artifacts reflect the 2.4.0 tool and are reproducible. See `scripts/regenerate_samples.sh`. (The annotated `.xlsx` content is reproducible but its openpyxl envelope carries wall-clock member mtimes, so it is not raw-byte identical — the `.html`/`.pdf` are.)
+
+## [2.4.0] — 2026-08-05
+
+### Fixed
+- **Mixed date formats are now detected in CSV files.** The CSV loader coerced date-like strings to `datetime` on load, erasing the raw format before the mixed-date analyzer could see it — so a column mixing `2026-01-01` and `01/02/2026` was silently reconciled and never flagged, the exact silent coercion datascope exists to surface. Date-like CSV cells are now kept as strings (a CSV has no type metadata; a date is text), so `analyze_mixed_dates` sees the raw formats and reports the inconsistency. Excel date cells, which arrive already typed from openpyxl, are unaffected.
+
+### Docs
+- README missing-value row now names both thresholds it depends on — flagged at ≥10% blank (`_DEFAULT_THRESHOLD_PCT`) and Warning at ≥50% / Info below (`findings/severity.py`). Correction: an earlier draft of this entry read "threshold now reads 10%", which put the flag floor into a row labeled *Warning* — the two are different decisions and the row now states both.
+- Removed the CSV mixed-date caveat from the README. The loader fix above now surfaces mixed dates in CSVs, so the caveat added in 2.3.4 ("supply as `.xlsx`") no longer holds.
+
 ## [2.3.4] — 2026-07-31
 
 ### Fixed

README.md

@@ -19,16 +19,14 @@ datascope finds these problems, explains what's wrong in plain English, and tell
 |---|---|---|
 | **Mixed types** | 485 numbers + 15 strings in a "numeric" column | Critical |
 | **Sentinel values** | "N/A", "TBD", "pending" hiding in numeric data | Critical |
-| **Missing values** | 50%+ of a column is blank — aggregations silently exclude those rows (below 50% is flagged Info) | Warning |
+| **Missing values** | Flagged at ≥10% blank; Warning at ≥50%, Info below — aggregations silently exclude those rows | Warning / Info |
 | **Leading-zero inconsistency** | "00123" alongside "456" — keys that won't match | Warning |
 | **Mixed date formats** | "01/15/2026" and "2026-01-15" in the same column | Warning |
 | **Suspected duplicate IDs** | 98% unique in an ID column — the other 2% will fan out joins | Warning |
 | **Near-constant columns** | 1 distinct value across 10,000 rows | Info |
 
 Each finding is expressed as **assumption vs. reality**: what the data *appears* to be vs. what it *actually contains*. Every finding includes a downstream impact explanation, a fix recommendation, and a prevention rule.
 
-> **Note on CSV input:** date strings written in formats the CSV loader recognizes (e.g. `2026-01-15` and `01/15/2026`) are parsed to real dates at load time, so a CSV column mixing those two formats is normalized before analysis and does **not** raise a *Mixed date formats* finding. To surface mixed date formats, supply the column as text — for example in an `.xlsx` file with text-formatted cells, where the values stay strings and the check fires as expected.
-
 ---
 
 ## Installation

datascope/__init__.py

@@ -2,4 +2,4 @@
 
 from __future__ import annotations
 
-__version__ = "2.3.4"
+__version__ = "2.4.0"

datascope/loaders/csv_loader.py

@@ -12,12 +12,10 @@
 from __future__ import annotations
 
 import csv
-from datetime import datetime
 from pathlib import Path
 
 import pandas as pd
 
-from datascope.analyzers.format_check import DATE_LIKE_RE
 from datascope.loaders.base import dedupe_headers
 from datascope.models import LoaderResult
 
@@ -30,17 +28,6 @@
 # it must stay a string rather than becoming ``inf``/``NaN``.
 _NON_FINITE_STRS = frozenset({"inf", "infinity", "nan"})
 
-# Date/time formats tried in order (most specific first).
-_DATETIME_FMTS = (
-    "%Y-%m-%dT%H:%M:%S",      # ISO 8601
-    "%Y-%m-%d %H:%M:%S",      # space-separated
-    "%Y-%m-%d",                # date only
-    "%m/%d/%Y %H:%M:%S",
-    "%m/%d/%Y",
-    "%d/%m/%Y",
-    "%Y/%m/%d",
-)
-
 
 def _infer_cell(raw: str) -> object:
     """Infer a single cell's Python value from its raw CSV string.
@@ -50,8 +37,14 @@ def _infer_cell(raw: str) -> object:
     2. Integer
     3. Float
     4. Boolean (true/false/yes/no, case-insensitive)
-    5. Datetime (common formats)
-    6. String fallback
+    5. String fallback
+
+    Date-like cells are DELIBERATELY left as strings. A CSV has no type
+    metadata, so a date is text; coercing it to ``datetime`` here would erase the
+    very format evidence the mixed-date analyzer needs, silently hiding a
+    mixed-format column ("2026-01-01" vs "01/02/2026") — exactly the silent
+    coercion datascope exists to surface. (Excel dates arrive already typed from
+    openpyxl, so the Excel loader keeps its datetime cells.)
     """
     stripped = raw.strip()
     if not stripped:
@@ -92,15 +85,10 @@ def _infer_cell(raw: str) -> object:
     if lower in _BOOL_FALSE:
         return False
 
-    # --- datetime -----------------------------------------------------
-    if DATE_LIKE_RE.match(stripped):
-        for fmt in _DATETIME_FMTS:
-            try:
-                return datetime.strptime(stripped, fmt)
-            except ValueError:
-                continue
-
     # --- string fallback ----------------------------------------------
+    # Date-like strings intentionally fall through to here (see docstring):
+    # kept as text so analyze_mixed_dates can see the raw format and flag a
+    # mixed-format column instead of it being silently coerced away.
     return stripped
 
 

datascope/reports/html.py

@@ -10,6 +10,7 @@
 import base64
 import datetime
 import html
+import os
 from collections import defaultdict
 from pathlib import Path
 from typing import Any
@@ -107,7 +108,16 @@ def write_html(
     filename = source_metadata.get("filename", "unknown")
     rows = source_metadata.get("row_count", "?")
     cols = source_metadata.get("column_count", "?")
-    now = datetime.datetime.now().strftime("%Y-%m-%d %H:%M")
+    # Honor SOURCE_DATE_EPOCH (reproducible-builds standard) so regenerated
+    # sample reports are byte-identical; a bare datetime.now() here makes the
+    # generator tag change every run and defeats any byte-lock on the output.
+    _sde = os.environ.get("SOURCE_DATE_EPOCH")
+    _dt = (
+        datetime.datetime.fromtimestamp(int(_sde), tz=datetime.timezone.utc)
+        if _sde
+        else datetime.datetime.now()
+    )
+    now = _dt.strftime("%Y-%m-%d %H:%M")
 
     counts = severity_counts(findings)
     total = sum(counts.values())

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "datascope-dq"
-version = "2.3.4"
+version = "2.4.0"
 description = "Data quality diagnostics for tabular datasets — surfaces hidden problems in plain English"
 readme = "README.md"
 requires-python = ">=3.10"

samples/output/sample_mixed_types_diagnostic.html

@@ -3,8 +3,8 @@
 <head>
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
-<meta name="description" content="Data quality diagnostic report for sample_mixed_types.xlsx — generated by datascope v2.3.4">
-<meta name="generator" content="datascope v2.3.4">
+<meta name="description" content="Data quality diagnostic report for sample_mixed_types.xlsx — generated by datascope v2.4.0">
+<meta name="generator" content="datascope v2.4.0">
 <link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><circle cx='50' cy='50' r='40' fill='%231f2e7a'/><text x='50' y='62' font-size='40' text-anchor='middle' fill='white' font-family='sans-serif' font-weight='bold'>d</text></svg>">
 <title>datascope diagnostic — sample_mixed_types.xlsx</title>
 <style>
@@ -92,7 +92,7 @@
 <div class="container">
   <div class="title-section">
     <h1>Data Quality Diagnostic</h1>
-    <div class="subtitle">sample_mixed_types.xlsx &middot; 200 rows &times; 4 columns &middot; 2026-07-31 21:18</div>
+    <div class="subtitle">sample_mixed_types.xlsx &middot; 200 rows &times; 4 columns &middot; 2026-08-05 00:00</div>
   </div>
 
   <div class="summary-row">
@@ -175,7 +175,7 @@ <h2>Field Inventory</h2>
   </div>
 
   <div class="footer">
-    Generated by datascope v2.3.4 &middot; 2026-07-31 21:18 &middot; 2 findings
+    Generated by datascope v2.4.0 &middot; 2026-08-05 00:00 &middot; 2 findings
     <br>pip install datascope-dq
   </div>
 </div>

samples/output/sample_sales_diagnostic.html

@@ -3,8 +3,8 @@
 <head>
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
-<meta name="description" content="Data quality diagnostic report for sample_sales.xlsx — generated by datascope v2.3.4">
-<meta name="generator" content="datascope v2.3.4">
+<meta name="description" content="Data quality diagnostic report for sample_sales.xlsx — generated by datascope v2.4.0">
+<meta name="generator" content="datascope v2.4.0">
 <link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><circle cx='50' cy='50' r='40' fill='%231f2e7a'/><text x='50' y='62' font-size='40' text-anchor='middle' fill='white' font-family='sans-serif' font-weight='bold'>d</text></svg>">
 <title>datascope diagnostic — sample_sales.xlsx</title>
 <style>
@@ -92,7 +92,7 @@
 <div class="container">
   <div class="title-section">
     <h1>Data Quality Diagnostic</h1>
-    <div class="subtitle">sample_sales.xlsx &middot; 500 rows &times; 15 columns &middot; 2026-07-31 21:18</div>
+    <div class="subtitle">sample_sales.xlsx &middot; 500 rows &times; 15 columns &middot; 2026-08-05 00:00</div>
   </div>
 
   <div class="summary-row">
@@ -243,7 +243,7 @@ <h2>Field Inventory</h2>
   </div>
 
   <div class="footer">
-    Generated by datascope v2.3.4 &middot; 2026-07-31 21:18 &middot; 5 findings
+    Generated by datascope v2.4.0 &middot; 2026-08-05 00:00 &middot; 5 findings
     <br>pip install datascope-dq
   </div>
 </div>

scripts/engagement_guard.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+"""Lailara engagement deploy guard (Python, stdlib-only).
+
+Exit 2 if an ACTIVE (non-demo) client engagement.yml is present in the current
+directory. No-op otherwise, so demo builds and clean CI checkouts are unaffected.
+Self-contained (no dependency on the installed lailara_engagement package) so it can
+run in any repo's deploy/build environment.
+"""
+import os
+import re
+import sys
+
+for _f in ("engagement.yml", "engagement.yaml"):
+    if os.path.isfile(_f):
+        with open(_f, encoding="utf-8-sig") as _fh:
+            _txt = _fh.read()
+        if re.search(r"^\s*demo:\s*true\s*$", _txt, re.M):
+            continue  # demo config -> safe
+        sys.stderr.write(
+            f"ENGAGEMENT GUARD: active client engagement config present ({_f}). "
+            "Client mode is runtime-only and must never deploy. Deactivate it "
+            "(set 'demo: true', or use engagement.demo.yml) before deploying.\n"
+        )
+        raise SystemExit(2)
+raise SystemExit(0)

scripts/git-hooks/pre-push

@@ -0,0 +1,24 @@
+#!/bin/sh
+# Lailara engagement deploy guard (git pre-push hook).
+#
+# Refuses to push while an ACTIVE (non-demo) client engagement.yml is present in
+# the working tree. Every tool repo auto-deploys on push, so blocking the push
+# blocks the deploy — client mode is runtime-only and must never ship.
+#
+# No-op when no engagement.yml exists (demo builds and clean CI checkouts push
+# normally), so demo behavior is unchanged.
+#
+# Activated per repo with:  git config core.hooksPath scripts/git-hooks
+set -e
+for f in engagement.yml engagement.yaml; do
+  if [ -f "$f" ]; then
+    if grep -Eq '^[[:space:]]*demo:[[:space:]]*true[[:space:]]*$' "$f"; then
+      continue   # demo config -> safe
+    fi
+    echo "ENGAGEMENT GUARD: active client engagement config present ($f)." >&2
+    echo "Client mode is runtime-only and must never deploy. Deactivate it" >&2
+    echo "(set 'demo: true', or remove/rename to engagement.demo.yml) before pushing." >&2
+    exit 2
+  fi
+done
+exit 0

scripts/regenerate_samples.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+# Regenerate the committed samples/output/ deliverables reproducibly.
+#
+# SOURCE_DATE_EPOCH pins the report generator timestamp (datascope/reports/html.py)
+# so the .html and .pdf outputs are byte-identical run-to-run. The value below is
+# 2026-08-05 00:00:00 UTC (the 2.4.0 release date); keep it stable so anyone
+# regenerating gets the same bytes, and bump it only on a release.
+#
+# Note: the annotated .xlsx is content-reproducible but NOT byte-lockable —
+# openpyxl stamps wall-clock times into the workbook envelope.
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+export SOURCE_DATE_EPOCH=1785888000  # 2026-08-05 00:00:00 UTC (2.4.0)
+
+for fmt in html pdf; do
+  python -m datascope samples/input/sample_mixed_types.xlsx --output-dir samples/output/ --format "$fmt"
+  python -m datascope samples/input/sample_sales.xlsx        --output-dir samples/output/ --format "$fmt"
+done
+python -m datascope samples/input/sample_mixed_types.xlsx --output-dir samples/output/ --format annotated-excel
+
+echo "Regenerated samples/output/ with SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"

tests/test_loaders.py

@@ -3,7 +3,6 @@
 from __future__ import annotations
 
 import textwrap
-from datetime import datetime
 from pathlib import Path
 
 import pytest
@@ -178,15 +177,25 @@ def test_boolean_inference(self, tmp_path):
         vals = list(result.dataframe["flag"])
         assert vals == [True, False, True, False]
 
-    def test_datetime_inference_iso(self, tmp_path):
+    def test_dates_stay_strings_so_mixed_formats_can_surface(self, tmp_path):
+        # P1 (RE-AUDIT): the CSV loader must NOT coerce date-like strings to
+        # datetime — that erased the format evidence and hid mixed-format columns
+        # (the silent coercion datascope exists to catch). Date-like CSV cells
+        # stay as strings; the mixed-date analyzer then sees the raw formats.
+        from datascope.analyzers.format_check import analyze_mixed_dates
+
         csv_path = _write_csv(tmp_path, """\
             ts
             2024-01-15
-            2024-06-30T12:00:00
+            01/15/2024
+            2024/06/30
         """)
         result = load_csv(csv_path)
-        assert all(t is datetime for t in result.cell_types["ts"])
-        assert result.dataframe.at[0, "ts"] == datetime(2024, 1, 15)
+        assert all(t is str for t in result.cell_types["ts"])
+        assert result.dataframe.at[0, "ts"] == "2024-01-15"   # not coerced
+        findings = analyze_mixed_dates(result)
+        assert len(findings) == 1                              # mixed formats now fire on CSV
+        assert set(findings[0].evidence["formats_found"]) == {"%Y-%m-%d", "%m/%d/%Y", "%Y/%m/%d"}
 
     def test_bom_handled_transparently(self, tmp_path):
         """UTF-8 BOM should not corrupt the first header."""