History rewrite: purge retired books, generated output, and legacy project blobs

[DataForSolution]

Why this issue exists

Current main has been cleaned through PR #10. Ordinary commits have removed generated output, redistributed third-party material, duplicate notebooks, local environments, and unreferenced binary assets from the live branch, but those objects may still exist in reachable Git history.

A one-time controlled history rewrite is therefore still required to make the repository materially smaller and to remove confirmed retired third-party/generated blobs from historical commits.

Current rewrite baseline

As of 2026-08-21 after PR #10:

• main: 6282f0af7591761b6922140d1b6bc9adba91be42
• GitHub-reported repository size: 1,118,991 KB (about 1.07 GiB)
• open pull requests: none
• the nine merged cleanup branch refs were force-moved to the current main commit, so they no longer point to earlier cleanup tips
• tag enumeration remains pending local mirror inspection
• deployment checks from Netlify/Vercel remain a separate pre-existing configuration issue

The remaining branch names are retained only for audit continuity; all observed cleanup branches now resolve to the same current main commit. During the local mirror preflight, verify every branch/tag ref before and after filtering.

Confirmed history-purge paths

Remove these paths from rewritten reachable history:

• assets/books/
• _site/
• _site - Copy/
• _projects-Tempo/
• _projects/
• projects/
• venv/
• flask_app/
• LLmachinPath.txt
• allfileshas.txt
• rebuild.md
• test.html
• test.md
• index-old.md
• assets/pdfs/2025-06-17-flexible-analytical-life-cycle.pdf
• toolkits/cheatsheets/
• toolkits/notebooks/

These are confirmed generated/local, redistributed third-party, temporary/duplicate, or superseded historical paths. Current main now uses canonical portfolio projects, official documentation, and publisher links instead of those copies.

Retired from live main, but preserve historical copies until archive/provenance classification

PR #10 removed these four unreferenced binaries from the live site and added exact ignore rules, but their historical objects must not be purged until they have been inspected and, if appropriate, archived independently:

• assets/documents/chest-cancer-detection.pdf
• assets/videos/background-video.mp4
• assets/videos/llmachine.mp4
• assets/toolkits/study_planners/ML_Study_Planner_60Days.pdf

No maintained page referenced them. Exact copies were not located in the connected Drive archive. Their absence from Drive is not evidence that they are disposable. Before filtering history, recover them from the untouched mirror, inspect provenance, and copy any original/valuable material to an independent archive or canonical project location.

Also preserve unless positively identified as obsolete:

• 176a0dc33af44f8289727353bf462244.txt — resembles an external verification token
• projects-new.html — intentionally redirects the legacy project route to /projects/

Required preflight

1. Current-branch cleanup: completed through PR Retire unreferenced binary assets from the live site #10.
2. Clone a fresh local mirror of the repository from GitHub.
3. Record main, every branch, every tag, and remote configuration.
4. Make an independent untouched mirror backup before rewriting.
5. From that mirror, enumerate historical paths and largest blobs.
6. Recover and classify/archive the four PR Retire unreferenced binary assets from the live site #10 binaries before deciding whether to add them to the purge command.
7. Capture repository size and largest-blob baseline. GitHub baseline recorded above; local mirror/object size still required.
8. Verify the Jekyll source builds from current main without committed generated output.
9. Confirm no credentials, private/local paths, redistributed material, or other unwanted data remain in reachable history.
10. Use git filter-repo; do not perform piecemeal history edits.
11. Re-clone the rewritten repository into a clean directory and validate it before force-pushing.
12. Coordinate every branch/tag ref; do not rewrite only main while stale refs preserve the old graph.
13. Keep the untouched rollback mirror until GitHub Pages and repository validation are complete.

Post-rewrite validation

• clean clone/fetch succeeds
• git fsck --full passes
• Jekyll source builds successfully
• GitHub Pages renders the expected site
• /projects/ shows only the curated ten-project portfolio
• /projects-new/ redirects to /projects/
• /ebooks/ remains the retirement/resource notice
• the Flexible Analytical Life Cycle post links to the publisher rather than hosting the publisher PDF
• /toolkits/ links to official documentation and the canonical Chest CT project
• every confirmed purge path is absent from rewritten reachable history
• archived/provenance-unknown assets are preserved independently before any historical removal
• cleanup branch/tag refs do not retain the pre-rewrite graph
• repository size is materially lower than the 1,118,991 KB GitHub baseline and the recorded local mirror baseline

Deployment integration note

Netlify and Vercel status checks are failing on cleanup PRs and were confirmed to be pre-existing rather than introduced by the latest content changes. Treat that as separate deployment-configuration debt. It does not replace local Jekyll build and clean-reclone validation for the history rewrite.

Coordination note

A history rewrite changes commit IDs and requires force-updating Git refs. Anyone with an existing clone must re-clone or deliberately reset afterward. Perform the rewrite from a verified local mirror/backup, not through GitHub file APIs.

[DataForSolution]

Local mirror preflight — read-only / no history rewrite yet

Run this from VS Code PowerShell in a non-OneDrive folder. Do not run git filter-repo yet.

# 1) Create an isolated maintenance workspace outside synced folders
$Work = 'C:\development\maintenance\dataforsolution-history'
New-Item -ItemType Directory -Force -Path $Work | Out-Null
Set-Location $Work

# 2) Fresh mirror from GitHub
git clone --mirror https://github.com/DataForSolution/DataForSolution.github.io.git source.git
if ($LASTEXITCODE -ne 0) { throw 'Mirror clone failed' }

# 3) Independent untouched rollback copy
Copy-Item -Path .\source.git -Destination .\rollback.git -Recurse

# 4) Enter the working mirror and record refs / tags / remote / object baseline
Set-Location .\source.git
git remote -v | Tee-Object ..\remote-before.txt
git for-each-ref --format='%(refname) %(objectname)' refs/heads refs/tags | Tee-Object ..\refs-before.txt
git count-objects -vH | Tee-Object ..\object-size-before.txt
git fsck --full | Tee-Object ..\fsck-before.txt

# 5) Confirm all branch heads currently converge on the cleaned main tip
$Expected = '6282f0af7591761b6922140d1b6bc9adba91be42'
git for-each-ref --format='%(refname:short) %(objectname)' refs/heads | Tee-Object ..\branches-before.txt
Write-Host "Expected cleaned main: $Expected"

# 6) Create an object/path inventory
# This can be large; keep it outside the mirror.
git rev-list --objects --all | Out-File -Encoding utf8 ..\all-objects-before.txt

# 7) If git-filter-repo is already available, run ANALYSIS ONLY
# This does not rewrite history.
git filter-repo --version
if ($LASTEXITCODE -eq 0) {
    git filter-repo --analyze
    Write-Host 'filter-repo analysis created under source.git\filter-repo\analysis'
} else {
    Write-Host 'git-filter-repo is not currently available; stop here and report that.'
}

# 8) Recover the four PR #10 binaries safely from the pre-removal commit
# Do this BEFORE any filtering. git archive preserves binary bytes.
$PrePR10 = '62b0201f79ca8901c31737201db92bf48e46441f'
git archive --format=zip --output=..\pending-assets-pre-pr10.zip $PrePR10 `
  assets/documents/chest-cancer-detection.pdf `
  assets/videos/background-video.mp4 `
  assets/videos/llmachine.mp4 `
  assets/toolkits/study_planners/ML_Study_Planner_60Days.pdf
if ($LASTEXITCODE -ne 0) { throw 'Pending-asset archive failed' }

Expand-Archive -Path ..\pending-assets-pre-pr10.zip -DestinationPath ..\pending-assets -Force
Get-ChildItem ..\pending-assets -File -Recurse |
  Get-FileHash -Algorithm SHA256 |
  Format-Table -AutoSize |
  Out-String -Width 240 |
  Tee-Object ..\pending-assets-sha256.txt

# 9) Return a concise preflight summary
Write-Host '--- OBJECT SIZE ---'
Get-Content ..\object-size-before.txt
Write-Host '--- REFS / TAGS ---'
Get-Content ..\refs-before.txt
Write-Host '--- PENDING ASSET HASHES ---'
Get-Content ..\pending-assets-sha256.txt

Stop condition

After the commands above, stop before any history rewrite or force push. Review:

• refs-before.txt
• object-size-before.txt
• fsck-before.txt
• pending-assets-sha256.txt
• source.git\filter-repo\analysis\ if available

Then classify/archive the four recovered binaries and review the largest historical paths/blobs. Only after that should the exact git filter-repo command be constructed and reviewed.

[DataForSolution]

Local rewrite gate complete; remote push held

Preflight, archival preservation, local filtering, and independent clean-clone validation are complete.

• GitHub main before rewrite: 26b9c1c3bff80f3f5b677099cc9fc3f0123419b7 (includes merged PR chore(site): make Jekyll validation reproducible #11 for reproducible Jekyll validation and one repaired local link)
• Locally rewritten main: de3f5c08029d30c0553e98c9f6a0abee1c9e1135
• packed size: 1.06 GiB before -> 13.32 MiB after
• confirmed purge path matches in rewritten reachable history: 0
• rewritten mirror git fsck --full: no reported errors
• independent clone/build: passed; 71 HTML pages; 2,781 local links checked; 0 broken
• required /projects/, /projects-new/, /ebooks/, /toolkits/, navigation, and publisher-link assertions: passed
• four pending historical assets: recovered independently with source commits, sizes, SHA-256 hashes, and provenance notes before filtering
• tags: none
• Netlify on PR chore(site): make Jekyll validation reproducible #11: passed
• Vercel on PR chore(site): make Jekyll validation reproducible #11: failed as pre-existing deployment configuration debt

Push blocker

GitHub currently exposes immutable refs/pull/*/head refs for PRs 1, 2, and 4-11. Several point to pre-rewrite commits. Ordinary branch/tag force-pushes cannot update these refs, so a force-push alone cannot guarantee that old objects become unreachable or that GitHub's repository-size metric materially drops.

Per the issue safeguards, no force-push has been performed. The untouched backup and external legacy-asset archive remain intact. Recommended next step is to coordinate GitHub Support/object cleanup, or explicitly accept the residual immutable-PR-ref/cached-object risk before the rewritten branch force-update.

[DataForSolution]

Remote history rewrite completed - Issue remains open for residual GitHub refs/objects

The authorized remote rewrite and fresh-GitHub-clone validation completed successfully.

Remote transition

• old remote main: 26b9c1c3bff80f3f5b677099cc9fc3f0123419b7
• new rewritten remote main: de3f5c08029d30c0553e98c9f6a0abee1c9e1135
• final pre-push tree equivalence: exact; both trees were 728c233d41ef5cd8a69b10ca4c243f359227b382
• open PRs before push: 0
• update method: one atomic push with an exact --force-with-lease guard for each writable branch
• writable branches updated: 11/11
• tags before/after: 0
• no refs/pull/* mutation was attempted

Size evidence

• local pack before: approximately 1.06 GiB
• local pack after: 13.32 MiB
• fresh GitHub clone .git size: 14,009,868 bytes (13.361 MiB)
• fresh GitHub clone packed objects: 13.32 MiB
• GitHub-reported repository size immediately after push: 1,118,992 KB

The GitHub-reported size has not yet shrunk. This is not treated as proof that the writable rewrite failed; immutable PR refs and GitHub server-side cached/uncollected objects remain.

Fresh GitHub clone validation

• fresh clone source: GitHub HTTPS remote, not the local rewritten mirror
• checked-out main: de3f5c08029d30c0553e98c9f6a0abee1c9e1135
• git fsck --full: passed with no reported output
• reachable confirmed purge-path matches across writable history: 0
• every remote writable branch exactly matches its corresponding rewritten branch: 11/11
• tags: none
• bundle exec jekyll build --trace: passed
• rendered HTML pages: 71
• internal links checked: 2,781
• broken internal links: 0
• /projects/: 10 curated project cards
• /projects-new/: redirects to /projects/
• /ebooks/: Learning Resources/retired-library notice preserved
• /toolkits/: canonical Chest CT project and official Python/reference links preserved
• Flexible Analytical Life Cycle article: publisher link present; local PDF reference absent
• PR chore(site): make Jekyll validation reproducible #11 Gemfile, Gemfile.lock ignore, and Career Hub Resume Reviewer link correction: present

Preserved historical assets

The untouched rollback mirror remains available. All four pending historical assets remain independently archived with their source commits, byte sizes, SHA-256 hashes, recovery packages, and provenance notes. The capstone PDF is classified as likely original work and preserved; both videos remain unresolved; the study planner is locally generated with authorship unresolved.

Deployment status

• Netlify PR chore(site): make Jekyll validation reproducible #11 deploy preview: success
• rewritten main Vercel status: failure
• Vercel is recorded as the known pre-existing deployment configuration debt and is not classified as a history-rewrite failure

Residual immutable GitHub pull refs

All 10 currently advertised refs/pull/*/head refs still point to commits absent from the rewritten graph and present in the untouched backup:

• refs/pull/1/head -> 4ea62061d4d8017a1ea537af4ca029ed6a5b163a
• refs/pull/2/head -> 312ed3b623137ea2cbe01d0d52662ccaf18ac5fc
• refs/pull/4/head -> cf63cc514171e074f6f2aa7d0f5787e94a75bd5d
• refs/pull/5/head -> d9167df3d72e96b452862d1c167d733b5db63943
• refs/pull/6/head -> 62d2dc2342abb0639af8ec376d5d5630ecb54e18
• refs/pull/7/head -> 6106ba49901ba5c9f9a119b37aaa19d09713c38f
• refs/pull/8/head -> 474fff3d9e272cac0f067167b1e1eaa3936726aa
• refs/pull/9/head -> 1f077dd587e08f664028502885b440eb8ed5d0ee
• refs/pull/10/head -> e12d1510a333f06f39a6b0ff2b09de421876c421
• refs/pull/11/head -> 83a5ac0c7ce6ea196b0fce48964a8a25a171441f

GitHub-managed PR refs and cached/server-side objects may therefore continue to retain and expose pre-rewrite objects. This issue must remain open.

Support assessment

No confirmed credential or other qualifying sensitive-data exposure was found. GitHub's published procedure states that Support does not remove non-sensitive data. A Support request solely for repository-size cleanup is therefore unlikely to result in PR-ref/cache removal and has not been opened. Repository deletion/recreation is not authorized and would require a separate strategic decision because it affects issues, PR history, Pages configuration, integrations, and repository metadata.