Skip to content

kolossi101/fragments

BranchesTags

Repository files navigation

Fragments Microservice

A scalable, cloud-native microservice for managing data fragments with secure REST API and automated CI/CD deployment.

Overview

The Fragments Microservice is a scalable backend service built on AWS to manage small pieces of data (“fragments”) such as text, JSON, CSV, YAML, Markdown, and images. It provides a secure REST API supporting full CRUD operations with JWT-based authentication, ensuring that data remains private and isolated per user. Binary data is stored in Amazon S3, while fragment metadata is managed in DynamoDB for fast retrieval and scalability. The service also supports on-the-fly format conversions (e.g., Markdown ➝ HTML) to reduce storage overhead. Deployment is automated with a CI/CD pipeline using GitHub Actions, Docker, ECS, and ECR, with testing handled via Jest and Hurl. This microservice is designed to power the Fragments Client App and demonstrate a modern, production-ready architecture for data management in distributed systems.

Features

  • Multi-Format Support: Text, JSON, CSV, YAML, Markdown, and image formats
  • Format Conversion: On-the-fly conversions (e.g., Markdown → HTML)
  • Secure Authentication: JWT-based auth with AWS Cognito
  • Cloud Storage: Binary data in S3, metadata in DynamoDB
  • RESTful API: Full CRUD operations with intuitive endpoints
  • Image Processing: Built-in image manipulation capabilities
  • Automated Testing: Unit and integration test suites
  • CI/CD Pipeline: Automated deployment with GitHub Actions
  • Containerized: Docker support for consistent environments
  • Monitoring: Structured logging with Pino

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Client Apps   │────│  Fragments API  │────│   AWS Services  │
│                 │    │                 │    │                 │
│ • Web App       │    │ • Express.js    │    │ • S3 (Storage)  │
│ • Mobile App    │    │ • JWT Auth      │    │ • DynamoDB      │
│ • CLI Tools     │    │ • Format Conv.  │    │ • Cognito       │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Technology Stack

  • Runtime: Node.js with Express.js v5.1.0 framework
  • Authentication: AWS Cognito with JWT verification, Passport.js
  • Storage: Amazon S3 for binary data, DynamoDB for metadata
  • Content Processing: Sharp (images), Markdown-it (markdown), js-yaml (YAML)
  • Testing: Jest (unit tests), Hurl (integration tests), Supertest (HTTP testing)
  • Deployment: Docker, AWS ECS, ECR
  • CI/CD: GitHub Actions
  • Logging: Pino structured logging

Prerequisites

Before running the Fragments Microservice, ensure you have:

  • Node.js (v16.0.0 or higher)
  • npm (v8.0.0 or higher)
  • AWS Account with appropriate permissions
  • Docker (optional, for containerized deployment)
  • curl and jq (for testing API endpoints)

AWS Services Required

  • AWS Cognito User Pool
  • Amazon S3 Bucket
  • DynamoDB Table
  • IAM Roles with appropriate permissions

Instructions on running the service

  1. Clone the repository

    git clone https://github.com/kolossi101/fragments
cd fragments

  2. Install dependencies

    npm install

  3. Set up environment configuration

    cp .env.example .env
# Edit .env with your configuration values

Configuration

Create a .env file in the project root with the following variables:

# Server Configuration
PORT=8080
NODE_ENV=development
LOG_LEVEL=debug

# AWS Configuration
AWS_REGION=us-east-1
AWS_COGNITO_POOL_ID=your_user_pool_id
AWS_COGNITO_CLIENT_ID=your_client_app_id

# Storage Configuration
AWS_S3_BUCKET_NAME=your-fragments-bucket
AWS_DYNAMODB_TABLE_NAME=fragments

# Authentication (for development/testing)
HTPASSWD_FILE=tests/.htpasswd

# Optional: API Configuration
API_URL=http://localhost:8080

Environment Variables

Variable Description Default Required
PORT Server port 8080 No
NODE_ENV Environment mode development No
LOG_LEVEL Logging level info No
AWS_COGNITO_POOL_ID Cognito User Pool ID - Yes
AWS_COGNITO_CLIENT_ID Cognito Client App ID - Yes
HTPASSWD_FILE Basic auth file path - No

Usage

Development Mode

Start the server with hot reload:

npm run dev

Production Mode

Start the server in production:

npm start

Debug Mode

Start with Node.js inspector enabled:

npm run debug

The server will be available at http://localhost:8080

API Documentation

Base URL

http://localhost:8080/v1

Authentication

The API supports two authentication methods:

  1. JWT Bearer Token (Production)

    Authorization: Bearer <jwt_token>

  2. Basic Authentication (Development)

    Authorization: Basic <base64_credentials>

Endpoints Overview

Method Endpoint Description
GET /fragments List user's fragments
GET /fragments/:id Get fragment data
GET /fragments/:id/info Get fragment metadata
POST /fragments Create new fragment
PUT /fragments/:id Update existing fragment
DELETE /fragments/:id Delete fragment

API Examples

Warning: First POST a fragment and paste into <FRAGMENT-ID> to work with GET method

1. Create a Text Fragment

curl -i \
  -X POST \
  -u user1@email.com:password1 \
  -H "Content-Type: text/plain" \
  -d "This is a fragment" \
  https://fragments-api.com/v1/fragments

Response:

{
  "status": "ok",
  "fragment": {
    "id": "30a84843-0cd4-4975-95ba-b96112aea189",
    "created": "2021-11-08T01:04:46.071Z",
    "updated": "2021-11-08T01:04:46.073Z",
    "ownerId": "11d4c22e42c8f61feaba154683dea407b101cfd90987dda9e342843263ca420a",
    "type": "text/plain",
    "size": 18
  }
}

2. Create a Markdown Fragment

curl -i \
  -X POST \
  -u user1@email.com:password1 \
  -H "Content-Type: text/markdown" \
  -d "# My Fragment\n\nThis is **bold** text." \
  https://fragments-api.com/v1/fragments

3. Create a JSON Fragment

curl -i \
  -X POST \
  -u user1@email.com:password1 \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello", "type": "greeting"}' \
  https://fragments-api.com/v1/fragments

4. Retrieve Fragment Data (GET /fragments/:id)

curl -i -u user1@email.com:password1 https://fragments-api.com/v1/fragments/{fragment-id}

5. Get Fragment Metadata

curl -i \
  -u user1@email.com:password1 \
  https://fragments-api.com/v1/fragments/{fragment-id}/info

6. List All Fragments

# Basic list
curl -s -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments" | jq

# Expanded list with metadata
curl -s -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments?expand=1" | jq

7. Update Fragment

curl -i -X PUT \
  -u user@example.com:password10 \
  -H "Content-Type: text/plain" \
  -d "Updated content here" \
  "http://localhost:8080/v1/fragments/{fragment-id}"

8. Delete Fragment

curl -i -X DELETE \
  -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments/{fragment-id}"

Supported Content Types

Type MIME Type Extensions Conversions
Text text/plain .txt -
Markdown text/markdown .md → HTML
HTML text/html .html -
JSON application/json .json -
YAML text/yaml .yaml, .yml -
CSV text/csv .csv -
Images image/png, image/jpeg, image/webp, image/gif Various Format conversion, resize

Fragment Structure

Each fragment consists of:

  • Metadata: Information about the fragment (id, type, size, dates)
  • Data: The actual binary content

Fragment Metadata:

{
  "id": "30a84843-0cd4-4975-95ba-b96112aea189",
  "ownerId": "11d4c22e42c8f61feaba154683dea407b101cfd90987dda9e342843263ca420a",
  "created": "2021-11-02T15:09:50.403Z",
  "updated": "2021-11-02T15:09:50.403Z",
  "type": "text/plain",
  "size": 256
}

API Examples

1. Create Fragments

Text Fragment:

curl -i -u user@example.com:password10 \
  -H "Content-Type: text/plain" \
  -d "This is a text fragment" \
  -X POST http://localhost:8080/v1/fragments | jq

Markdown Fragment:

curl -i -u user@example.com:password10 \
  -H "Content-Type: text/markdown" \
  -d "# My Fragment\n\nThis is **bold** text." \
  -X POST http://localhost:8080/v1/fragments | jq

JSON Fragment:

curl -i -u user@example.com:password10 \
  -H "Content-Type: application/json" \
  -d '{"coffee": "macchiato", "size": "large"}' \
  -X POST http://localhost:8080/v1/fragments | jq

Response (HTTP 201):

{
  "status": "ok",
  "fragment": {
    "id": "30a84843-0cd4-4975-95ba-b96112aea189",
    "ownerId": "11d4c22e42c8f61feaba154683dea407b101cfd90987dda9e342843263ca420a",
    "created": "2021-11-02T15:09:50.403Z",
    "updated": "2021-11-02T15:09:50.403Z",
    "type": "text/plain",
    "size": 256
  }
}

2. Retrieve Fragments

Get Fragment Data:

curl -i -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments/{fragment-id}"

# Returns raw binary data with appropriate Content-Type header

Get Fragment Metadata:

curl -i -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments/{fragment-id}/info" | jq

3. List Fragments

Basic List (IDs only):

curl -i -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments" | jq

# Response:
{
  "status": "ok",
  "fragments": [
    "4dcc65b6-9d57-453a-bd3a-63c107a51698",
    "30a84843-0cd4-4975-95ba-b96112aea189"
  ]
}

Expanded List (Full Metadata):

curl -i -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments?expand=1" | jq

# Returns array of complete fragment metadata objects

4. Update Fragment

curl -i -X PUT \
  -u user@example.com:password10 \
  -H "Content-Type: text/plain" \
  -d "This is updated data" \
  "http://localhost:8080/v1/fragments/{fragment-id}"

# Note: Content-Type must match the original fragment type

5. Delete Fragment

curl -i -X DELETE \
  -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments/{fragment-id}"

# Response (HTTP 200):
{ "status": "ok" }

Supported Content Types & Formats

Type MIME Type Extensions Description
Text text/plain .txt Plain text content
Markdown text/markdown .md Markdown formatted text
HTML text/html .html HTML markup
CSV text/csv .csv Comma-separated values
JSON application/json .json JSON data
YAML application/yaml .yaml, .yml YAML data
PNG image/png .png PNG images
JPEG image/jpeg .jpg JPEG images
WebP image/webp .webp WebP images
AVIF image/avif .avif AVIF images
GIF image/gif .gif GIF images

Format Conversions

Fragments can be converted to different formats by appending file extensions to the fragment ID:

Valid Conversions

Original Type Available Conversions
text/plain .txt
text/markdown .md, .html, .txt
text/html .html, .txt
text/csv .csv, .txt, .json
application/json .json, .yaml, .yml, .txt
application/yaml .yaml, .txt
All Images .png, .jpg, .webp, .gif, .avif

Conversion Examples

# Get Markdown fragment as HTML
curl -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments/markdown-id.html"

# Get JSON as YAML
curl -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments/json-id.yaml"

# Convert image formats
curl -u user@example.com:password10 \
  "http://localhost:8080/v1/fragments/image-id.png"

Error Codes

Code Description
400 Bad Request - Invalid request format or data
401 Unauthorized - Authentication required
404 Not Found - Fragment doesn't exist or doesn't belong to user
415 Unsupported Media Type - Invalid content type or conversion
500 Internal Server Error - Server-side error

Security & Ownership

  • All fragments are private and isolated per authenticated user
  • Users can only access their own fragments
  • Fragment ownership is determined by the ownerId (SHA256 hash of email)
  • No public access to any fragment data

Testing

The project includes comprehensive testing suites:

Available Test Commands

# Run all unit tests
npm test

# Run tests in watch mode
npm run test:watch

# Generate coverage report
npm run coverage

# Run integration tests
npm run test:integration

# Run linting
npm run lint

Test Structure

tests/
├── unit/           # Jest unit tests
├── integration/    # Hurl integration tests
├── images/         # Test image data
└── .htpasswd      # Basic auth credentials for testing

Writing Tests

Unit tests use Jest and should be placed in the tests/unit/ directory:

// Example test
describe('Fragment', () => {
  test('should create valid fragment', () => {
    const fragment = new Fragment({
      ownerId: 'user123',
      type: 'text/plain',
    });
    expect(fragment.type).toBe('text/plain');
  });
});

Deployment

Docker Deployment

  1. Build Docker image

    docker build -t fragments-microservice .

  2. Run container

    docker run -p 8080:8080 --env-file .env fragments-microservice

AWS ECS Deployment

The service includes automated deployment via GitHub Actions:

  1. Set up GitHub Secrets:

    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • AWS_REGION
    • Other environment variables

  2. Deploy: Push to main branch triggers automatic deployment

Manual AWS Setup

  1. Create S3 bucket for fragment storage
  2. Set up DynamoDB table for metadata
  3. Configure Cognito User Pool for authentication
  4. Deploy to ECS using provided task definition

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Development Guidelines

  • Follow ESLint configuration
  • Write tests for new features
  • Update documentation as needed
  • Ensure all tests pass before submitting PR

Troubleshooting

Common Issues

Port already in use

# Find process using port 8080
lsof -i :8080
# Kill the process
kill -9 <PID>

AWS credentials not found

  • Ensure AWS CLI is configured: aws configure
  • Or set environment variables in .env

Fragment not found

  • Check fragment ID is correct
  • Verify user authentication
  • Ensure fragment belongs to authenticated user

Docker build fails

  • Check Docker daemon is running
  • Verify all required files are present
  • Review Dockerfile for syntax errors

Debug Mode

Enable debug logging:

LOG_LEVEL=debug npm run dev

Health Check

Check service health:

curl http://localhost:8080/

Logs

View application logs:

# In development
npm run dev

# In Docker
docker logs <container-id>

About

Back-end microservice using AWS cloud technologies.

youtu.be/VhwVPmxk7L4

Topics

aws microservice

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

10 tags

Packages

 
 
 

Contributors

Languages