A production-ready web application that connects to multiple databases, extracts and edits data in batches, and stores results securely — built with Next.js, Django REST Framework, and Docker.
| Layer | Technology | Why |
|---|---|---|
| Frontend | Next.js 13 (App Router) | Server-side rendering, file-based routing, and excellent TypeScript support make it ideal for data-heavy dashboards |
| Backend | Django REST Framework | Batteries-included framework with built-in ORM, serializers, and authentication — reduces boilerplate significantly |
| Auth | JWT (SimpleJWT) | Stateless authentication that scales well; no server-side session storage needed |
| Primary DB | PostgreSQL | ACID-compliant, battle-tested relational DB for storing app data and user records |
| Connectors | PostgreSQL, MySQL, MongoDB, ClickHouse | Covers the most common relational, document, and analytical database types |
| Containerization | Docker + Docker Compose | Ensures identical environments across development and production; single command to spin up all services |
| Styling | Tailwind CSS + shadcn/ui | Utility-first CSS with accessible, composable components |
Each database type implements a common interface with test_connection() and extract_data(query, batch_size) methods. This makes adding new database types trivial — just create a new class that implements the interface.
connectors/
├── base.py # Abstract base class
├── postgresql.py # PostgreSQL implementation
├── mysql.py # MySQL implementation
├── mongodb.py # MongoDB implementation
└── clickhouse.py # ClickHouse implementation
Why this approach? The Strategy pattern allows the system to be extended without modifying existing code (Open/Closed Principle). Adding a new DB type like SQLite or Redis requires zero changes to existing connectors.
When data is submitted, it is saved in two places simultaneously:
- PostgreSQL: structured records for querying and auditing
- JSON/CSV file: portable export with timestamp and source metadata for data portability
Why both? The database provides fast querying and relational integrity. The file export provides portability — users can download and use their data outside the platform.
JWT was chosen over session-based authentication because:
- Stateless — Django doesn't need to store session data
- Works seamlessly with Next.js API calls
- Access + refresh token pattern balances security and UX
Two roles are implemented:
- Admin: full access to all connections, extractions, and files
- User: access only to their own files and files explicitly shared with them
Permissions are enforced at the API level (Django), not just the frontend, preventing unauthorized access even via direct API calls.
All 6 services (frontend, backend, PostgreSQL, MySQL, MongoDB, ClickHouse) run in isolated containers with a shared network. The backend waits for PostgreSQL to be healthy before starting, preventing race conditions.
full-stack-Data-Connector-Platform/
├── frontend/ # Next.js application
│ ├── app/ # App Router pages
│ ├── components/ # Reusable UI components
│ ├── contexts/ # React context providers
│ ├── hooks/ # Custom React hooks
│ └── lib/ # Utilities and API clients
├── backend/ # Django application
│ ├── api/ # Models, views, serializers, URLs
│ ├── connectors/ # DB connector abstraction layer
│ ├── dataconnector/ # Django settings and config
│ └── manage.py
├── docker-compose.yml # All services definition
└── README.md
- Docker Desktop installed and running
- Git
git clone https://github.com/Judiciousmurich/full-stack-data--connector-platform.git
cd full-stack-data--connector-platformdocker compose up --buildThis will start:
- Frontend: http://localhost:3000
- Backend API: http://localhost:8000
- PostgreSQL: port 5432
- MySQL: port 3306
- MongoDB: port 27017
- ClickHouse: port 8123
docker compose exec backend python manage.py shellfrom api.models import User
u = User(email='admin@gmail.com', role='admin', is_staff=True, is_superuser=True)
u.set_password('YourPassword@123')
u.save()
exit()Go to http://localhost:3000 and sign in with your admin credentials.
| Method | Endpoint | Description | Auth |
|---|---|---|---|
| POST | /api/auth/register/ |
Register new user | No |
| POST | /api/auth/login/ |
Login, returns JWT | No |
| GET | /api/auth/me/ |
Current user profile | Yes |
| POST | /api/auth/token/refresh/ |
Refresh JWT token | No |
| GET/POST | /api/connections/ |
List / create DB connections | Yes |
| DELETE | /api/connections/<id>/ |
Delete a connection | Yes |
| POST | /api/connections/<id>/test/ |
Test a DB connection | Yes |
| POST | /api/extract/ |
Extract data from a connection | Yes |
| POST | /api/grid/submit/ |
Submit edited rows (dual storage: external DB + file) | Yes |
| GET/POST | /api/files/ |
List / upload files | Yes |
| GET | /api/files/<id>/download/ |
Download a file | Yes |
| GET | /api/activity/ |
Recent activity log | Yes |
| GET | /api/dashboard/stats/ |
Dashboard statistics | Yes |
| GET/POST | /api/admin/users/ |
List / create users (admin only) | Admin |
| DELETE | /api/admin/users/<id>/ |
Delete a user (admin only) | Admin |
docker compose exec backend python manage.py test api.testsExpected output:
Ran 10 tests in 9.273s
OK
- Multi-database connector: connect to PostgreSQL, MySQL, MongoDB, and ClickHouse with an extensible abstraction layer
- Batch data extraction: pull data from any configured source with configurable batch sizes
- Editable data grid: inline editing with row updates and basic validation
- Dual storage: submitted data saved to both database and JSON/CSV files with timestamps
- Role-based access control: admin full access; users see only their own files
- Containerized infrastructure: full Docker Compose setup for all services
MIT