Skip to content

Repository files navigation

Data Reclaim

Take back your data. Turn a Google Takeout export into searchable, queryable data with source-cited answers — entirely on your own machine. No cloud, no telemetry, no third-party AI. Your data never leaves your computer.

"Google Takeout" is Google's data-export feature (Settings → Takeout) — not food. It hands you your account data as hundreds of unreadable JSON files: "we gave you everything," technically compliant and practically useless. Data Reclaim makes it actually yours.

(The DOCX converter component is still called FireConverterfireconverter.py.)

There are two paths, for two goals:

  • Read your data → fireconverter.py + book_compiler.py produce Word documents / a merged "book." Good for small and medium files.
  • Use / query your data → reclaim_indexer.py produces a local SQLite database and JSONL exports. Ask questions, count things, filter, and feed selected slices to an AI — locally.

Honest positioning (read this)

As yet another JSON→DOCX converter, this does nothing special — there are dozens, and the tech (json + python-docx) is textbook. There is no patentable novelty here and the code claims none. Copyright in the source is automatic; that's the only IP that applies.

What makes this worth keeping is the query path and the privacy stance. A Google Takeout can hold 1–2 million records in a single file (Location History). Rendering that to Word is pointless — nobody reads a million rows, and the file won't even open. The indexer turns the same data into something you can interrogate:

  • Measured: the indexer ingests 200,000 records in ~2.7s (~75,000/sec) and scales to the giant files. The DOCX converter, by contrast, needed ~99s for 10,000 records and produced unopenable output on the giants. The query path is roughly 740× faster per record and doesn't collapse.
  • Zero dependencies: reclaim_indexer.py uses only the Python standard library.
  • Your "how much data" point, quantified: SELECT COUNT(*) tells you exactly how many location pings / searches / watched videos Google logged on you.

The DOCX path is kept because reading a document is genuinely nicer for small files — but for the archive as a whole, the query path is the point.

Install & usage

Full guide: INSTALL_AND_USAGE.md. Quick version:

pip install -r requirements.txt        # only for the DOCX read path

# Query path (recommended)
python3 reclaim_indexer.py --input-dir /path/to/Takeout --db reclaim.db
sqlite3 reclaim.db "SELECT data_type, SUM(n_records) FROM sources GROUP BY data_type ORDER BY 2 DESC;"
python3 reclaim_answer.py ask --db reclaim.db "python videos I watched" --limit 10               # cited passages
python3 reclaim_answer.py ask --db reclaim.db "feeling anxious" --mode semantic --limit 10         # cited, by concept

# Read path
python3 fireconverter.py --input data.json --output data.docx
python3 book_compiler.py --input-dir /path/to/Takeout --output book.docx

# One-command Takeout -> book
./run_reclaim.sh /path/to/Takeout /path/to/book.docx

Using with AI agents

Both Claude and Hermes use the SKILL.md format (compatible with the agentskills.io open standard). This repo ships a portable skill at skills/data-reclaim/ and doubles as a Hermes tap (hermes skills tap add QuietFireAI/data-reclaim); data-reclaim.skill is the Claude-packaged version. Gemini does not use SKILL.md — it uses function calling, wired in integrations/ (shared tested dispatch.py, tools_schema.json, reference loops). Gemini is cloud, so passages sent back to it leave your machine. See AGENTS.md.

Testing

pip install -r requirements.txt
python -m unittest discover -s tests -v

15 end-to-end tests exercise the real CLIs: DOCX structural validity (exactly one trailing sectPr), volume-splitting, book merge, indexing counts/shapes, full-text search, cited answers (relevance reported higher = better), the no-match refusal, and cite round-trip. The optional semantic test auto-skips if fastembed isn't installed. CI runs the suite on Python 3.9/3.11/3.12 (see .github/workflows/ci.yml).

Takeout unpacking

The numbered ZIPs are parts of one archive. Unzip them all into one folder so the structure rebuilds:

mkdir -p /path/to/Google-Takeout-Master
find /path/to/zips -name '*.zip' -exec unzip -o {} -d /path/to/Google-Takeout-Master \;

What was fixed / added vs the original scripts

Found by testing, not by reading:

  1. book_compiler produced structurally invalid DOCX (stranded <w:sectPr> markers; failed Microsoft's XSD schema). Fixed: deep-copy content, skip source section markers, insert before the master's trailing sectPr.
  2. book_compiler broke at scale — passing every path on the command line exceeds the OS ARG_MAX limit on a real Takeout. Added --input-dir / --manifest.
  3. fireconverter collapsed on large files (super-linear slowdown, unopenable output). Added volume-splitting.
  4. Added reclaim_indexer.py — the query path (SQLite + JSONL), the actual reason this repo earns its space.

On vector databases (honest take)

A vector database (embeddings + semantic search) would let you ask fuzzy questions like "when was I researching X." It's a real upgrade for the text-heavy data only — search history, YouTube history, activity text. It is useless for location/numeric telemetry: embedding latitude/longitude pairs produces nonsense, and those files are the bulk of the volume. A vector layer also needs a local embedding model (a heavy dependency and real compute) to stay private.

Recommendation: the SQLite index is the correct foundation and ships now. A local vector index over the text tables is a sensible optional Phase 2 — but only if you want semantic Q&A, and it should be built on top of the structured data, not instead of it. Not built yet.

Roadmap

  • Local vector search over text tables — shipped (reclaim_semantic.py, and reclaim_answer.py --mode semantic).
  • Possible: incremental re-indexing, and a small TUI for querying.

Support & contact

License

MIT — see LICENSE. Copyright (c) 2025 Jeff Phillips.

Part of the open-source collection at https://github.com/QuietFireAI

About

Take back your data. Turn your data file from an AI (Google, Facebook, OpenAI) export into searchable, queryable data with source-cited answers — entirely on your own machine. No cloud, no telemetry, no third-party AI. Your data never leaves your computer.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages