"Mini" Lakehouse Implementation

A local data lakehouse implementation built with data engineering tools, designed to mirror real-world pipeline architecture at a small scale.

Contains both raw (unprocessed) and structured (optimized for querying) data.

---

Why "mini"?

"Mini" because to be a proper Lakehouse many more features would have to be present, like for instance ACID transactions, workflow orchestration, partitioning at scale, etc.

I intend to create a more complex and fully featured Data Lakehouse later, but for now this was mostly for implementation practice.

---

Architecture

The pipeline follows the Medallion Architecture, structured in three layers:

• Bronze — raw data as received from the source (JSON), stored in S3 with no transformations. Permanent record of every fetch.
• Silver — cleaned, typed, and flattened data written to S3 as Parquet. Optimized for analytical processing.
• Gold — aggregated data loaded into PostgreSQL tables, ready for querying and consumption.

Data Workflow

API → S3/bronze/ (JSON) → S3/silver/ (Parquet) → PostgreSQL (Gold)

---

Softwares Used

Tool	Role
Docker + Compose	Containerization
Terraform	Infrastructure provisioning
LocalStack	Local AWS S3 emulation
PostgreSQL	Gold layer
Python + pandas	Data transformation
uv	Python package management

---

Prerequisites

• Docker + Compose
• Terraform
• AWS CLI
• uv
• Makefile (if you don't have Make, you can also just see the commands in the Makefile and execute them yourself, but i strongly recommend using Make)

---

Setup

In case you want to test it, it's fairly straight forward.

1. Clone the repository

    git clone https://github.com/benaytms/mini-lakehouse.git
    cd mini-lakehouse

2. Create .env from example

   cp .env.example .env
   # open .env and fill in the variables

3. Configure Terraform variables

   cp terraform/terraform.tfvars.example terraform/terraform.tfvars
   # open terraform.tfvars and set your bucket name
   # must match BUCKET_NAME in .env

4. Run the pipeline

   make up     # starts containers and S3 bucket
   make run    # runs the full pipeline (Bronze->Silver->Gold)

5. Explore the data

   make psql

   Once inside, try:

   SELECT * FROM countries LIMIT 10;
   SELECT * FROM covid_cases LIMIT 10;

   Exit

   \q
   

6. Teardown

   make down    # clear bucket files, stops all containers
   docker container prune     # Removes all stopped containers 
                              # CAREFUL in case you have other stopped containers!)

---

License

MIT