Laravel External Data Import Application

This repository contains a Laravel application designed to fetch, process, and store user and post data from external APIs (DummyJSON or JSONPlaceholder) efficiently.

Setup and Running the Application

Follow these steps to set up the project locally:

1. Clone the repository

   git clone https://github.com/MHJanny/laravel-external-data.git
   cd laravel-external-data

2. Install Dependencies

   # Install PHP dependencies
   composer install
   
   # Install Node.js dependencies
   npm install
   
   # Compile front-end assets
   npm run build

3. Environment Configuration

   Copy the example environment file.

   cp .env.example .env

4. Generate Application Key

   php artisan key:generate

5. Run Migrations

   Create the necessary database tables.

   php artisan migrate

6. Run the Import Command

   Execute the custom Artisan command to fetch and import data.

   php artisan app:import-data

7. Start the Application

   php artisan serve

Technical Approach

Large Data Fetching, Storage, and Optimization

To handle potentially large datasets and ensure system stability, the following strategies were implemented:

1. Generators for Memory Efficiency:

   • Instead of loading all API data into memory at once, I utilized PHP Generators (yield). The DummyJsonService and JsonPlaceholderService fetches data in pages (pagination) and yields chunks of data. This allows the application to process millions of records with constant low memory usage.

2. Batch Processing & Database Upserts:

   • Data is processed in batches (chunks).
   • I used User::upsert() and Post::upsert() instead of individual create or update calls. This significantly reduces database round-trips and improves write performance.
   • upsert also handles idempotency, ensuring that re-running the command updates existing records rather than creating duplicates.

3. Resilient API Communication:

   • The HTTP client is configured with retry logic (3 retries with exponential backoff) and timeouts (connect and request) to handle network instability or API rate limiting gracefully.

4. Abstraction & Extensibility:

   • A DataSourceInterface was created to define the contract for data fetching.
   • The AppServiceProvider binds the concrete implementation based on configuration. This makes it trivial to swap data sources (e.g., switching from DummyJSON to JSONPlaceholder) without changing the core import logic.

Challenges & Solutions

1. Memory Exhaustion with Large Datasets

• Challenge: Fetching thousands of records at once can easily exceed PHP's memory limit.
• Solution: Implemented pagination in the API service combined with PHP Generators. The ImportDataCommand iterates over the generator, processing one batch at a time, ensuring memory usage remains stable regardless of the dataset size.

2. API Rate Limiting and Network Flakiness

• Challenge: External APIs can fail intermittently or block requests if sent too quickly.
• Solution: Used Laravel's HTTP Client retry() method to automatically retry failed requests. Added sleep intervals between retries to respect potential rate limits.

3. Data Integrity & Duplicates

• Challenge: Running the import command multiple times could result in duplicate data or outdated records.
• Solution: Utilized upsert (Update or Insert) based on the unique external_id. This ensures that the local database always reflects the latest state of the external source without duplication errors.