diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..2bdb47c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,26 @@ +--- +name: Bug Report +about: Create a report to help us improve +title: '[BUG] ' +labels: bug +assignees: '' +--- + +## Describe the Bug +A clear and concise description of what the bug is. + +## Expected Behavior +A clear and concise description of what you expected to happen. + +## Screenshots +If applicable, add screenshots to help explain your problem. + +## Environment +- Package: [core/server/cli/dashboard/client] +- Version: [e.g., 1.0.0] +- OS: [e.g., Windows/Mac/Linux] +- Node Version: [e.g., 18.x] +- Browser: [e.g., Chrome, Safari] + +## Additional Context +Add any other context about the problem here. \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..23b7064 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,27 @@ +--- +name: Feature Request +about: Suggest an idea for this project +title: '[FEATURE] ' +labels: enhancement +assignees: '' +--- + +## Is your feature request related to a problem? +A clear and concise description of what the problem is. + +## Describe the solution you'd like +A clear and concise description of what you want to happen. + +## Describe alternatives you've considered +A clear and concise description of any alternative solutions or features you've considered. + +## Which package does this affect? +- [ ] Core +- [ ] Frontend Dashboard +- [ ] Cli Dashboard +- [ ] Serve +- [ ] Client +- [ ] Documentation + +## Additional context +Add any other context or screenshots about the feature request here. \ No newline at end of file diff --git a/docs/architecture.md b/docs/architecture.md index 1d34ebc..0c41b22 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,16 +1,164 @@ # JetQueue Architecture -## Packages +## Overview -- `@jet-queue/core` – engine (events, retries, storage) -- `@jet-queue/server` – Bun server (REST + WebSocket) -- `@jet-queue/client` – SDK for any environment -- `@jet-queue/cli` – terminal dashboard -- `@jet-queue/dashboard` – web dashboard (Next.js) +JetQueue is a modular background job processing system built with TypeScript. It consists of five core packages that work together to provide a complete job queue solution, from task definition to execution monitoring. + +--- + +## Package Structure + +### `@jet-queue/core` + +The heart of JetQueue – a standalone queue engine that handles all job lifecycle management. + +- **Job lifecycle management**: Create, track, update, and remove jobs +- **Retry logic**: Fixed, linear, and exponential backoff strategies +- **Storage abstraction**: In-memory by default, with SQLite persistence (built into Bun/Node.js) +- **Event system**: Emits typed events for every job state change +- **Priority queue**: Four priority levels (critical > high > normal > low) +- **Concurrency control**: Configurable worker pool size + +**Dependencies**: None (pure TypeScript) + +--- + +### `@jet-queue/server` + +A ready-to-use Bun server that wraps the core engine with REST and WebSocket APIs. + +- **REST endpoints**: Job submission, retrieval, cancellation, retry, stats, and health checks +- **WebSocket server**: Real-time job events (start, complete, fail, progress, stats updates) +- **Built-in SQLite persistence**: Jobs persist between server restarts (using Bun/Node.js SQLite) +- **Ready to run**: Zero-configuration setup for development and production +- **Customizable**: Developers can extend or replace with their own server implementation + +**Dependencies**: `@jet-queue/core`, Bun runtime + +--- + +### `@jet-queue/client` + +TypeScript/JavaScript SDK for interacting with JetQueue servers. + +**Capabilities**: + +- Submit jobs with full configuration (priority, delay, retry, timeout, tags, metadata) +- Subscribe to job events via WebSocket (progress, completion, failure, etc.) +- Fetch job status, stats, and list jobs +- Cancel or retry existing jobs +- Type-safe API with full TypeScript support +- Works in any environment: Node.js, Bun, browsers, edge functions + +**Dependencies**: None (pure TypeScript) + +--- + +### `@jet-queue/cli` + +Terminal-based dashboard for server management. + +**Features**: + +- View queue stats (pending, running, completed, failed, delayed, total) +- List and filter jobs by status or tags +- Inspect job details (data, metadata, error messages) +- Cancel or retry jobs directly from the terminal +- Monitor real-time job progress +- Connect to local or remote servers + +**Dependencies**: `@jet-queue/client` + +--- + +### `@jet-queue/dashboard` + +Web-based administrative UI built with Next.js. + +**Features**: + +- Real-time job monitoring with auto-refresh +- Visual queue statistics and charts +- Job search and filtering (by status, tags, name) +- Detailed job inspection with expandable data/error views +- Manual job management (cancel, retry, add) +- Responsive design for desktop and mobile + +**Dependencies**: `@jet-queue/client`, Next.js + +--- + +## Core Domain Model + +### Job States + +```typescript +type JobStatus = + | "pending" // Waiting in queue + | "running" // Currently executing + | "completed" // Finished successfully + | "failed" // Failed (will stay in this state until manual retry) + | "delayed" // Scheduled for future execution + | "cancelled"; // Manually cancelled +``` + +### Job Configuration + +Each job supports: + +- **Priority**: `critical` (0) > `high` (1) > `normal` (2) > `low` (3) +- **Retry strategies**: Fixed, linear, or exponential backoff +- **Timeout**: Per-job execution time limit +- **Delay**: Schedule for future execution +- **Max attempts**: Retry limit before marking as failed +- **Tags & Metadata**: For filtering and custom data ## Data Flow -```text -App → client.addJob() → HTTP POST /api/jobs → server → queue.add() → process -App ← client.onJobCompleted() ← WebSocket ← server ← queue.emit('job:completed') +```mermaid +sequenceDiagram + participant Client as Client
(SDK/App) + participant Server as Server
(Bun) + participant Core as Core Engine + participant DB as SQLite Database + + Client->>Server: HTTP POST /api/jobs + Server->>Core: Process job + Core->>DB: Read/Write data + + loop Real-time Updates + Core-->>Client: WebSocket Events
job:completed, job:failed,
job:progress, stats, etc. + DB-->>Client: Events + end ``` + +### Step-by-Step Flow + +1. Job Submission + - Client calls `client.addJob()` with job data and options + - HTTP request sent to `POST /api/jobs` + - Server validates and passes job to Core engine + - Core stores job in SQLite and adds to queue + - Server returns job ID to client +2. Job Processing + - Core engine picks next job based on priority and concurrency limits + - Job status transitions: pending → running + - Task function executes with configured timeout + - Progress updates can be emitted during execution +3. Completion + - Success: Status → completed, result stored + - Failure: Status → failed, error stored + - All state changes persisted to SQLite +4. Real-time Updates + - Core emits events for every state change + - Server broadcasts events via WebSocket to connected clients + - Clients receive typed events in real-time + +## Design Principles + +1. **Modular** – Each package can be used independently +2. **Simple** – Zero-config defaults, easy to extend +3. **Type-safe** – Full TypeScript with strict typing +4. **Event-driven** – Real-time updates for all state changes +5. **Persistent by default** – SQLite ensures jobs survive restarts +6. **Developer-first** – Clear API, excellent dev experience \ No newline at end of file diff --git a/docs/getting-started.md b/docs/getting-started.md index 03aa5ef..fe86ff7 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -4,6 +4,14 @@ ```bash npm install @jet-queue/core + +# or + +pnpm add @jet-queue/core + +# or + +yarn add @jet-queue/core ``` ## Your First Queue @@ -30,6 +38,8 @@ queue.on("queue:drain", () => { ```bash npx @jet-queue/server --port 3001 ``` +> [!TIP] +> you can use whatever package manger you want, here is the npm example Then connect with the client: diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..8da69be --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,27 @@ +# JetQueue - Background job processor +**Keep your API snappy. Process heavy tasks in the background.** + +> JetQueue moves time-consuming operations out of your request/response cycle, giving users sub-100ms response times while emails send, PDFs generate, and data processes in parallel. + + +## Features +- **Concurrent Processing** - Run N jobs simultaneously +- **Auto-Retry** - Exponential, linear, or fixed backoff strategies +- **Priority Queues** - Critical jobs jump the line +- **Delayed & Recurring Jobs** - Schedule work for later + +## **Where** and **Why** you should use it + +> Be decisive can use it every where that you are working with asynchronous tasks and queues + +| Industry | Examples | +|---|---| +| **E-commerce** | Order confirmation emails, inventory sync, invoice PDF generation | +| **Social Media** | Image/video processing, notification dispatch, feed generation | +| **SaaS** | Report generation, data exports, search indexing, billing | +| **AI/ML** | Model inference, data preprocessing, batch predictions | +| **IoT** | Sensor data processing, device command queues | + +## Examples and demo + +*For better understanding of what does actually JetQueue do check out our examples [here](./examples.md)* \ No newline at end of file diff --git a/docs/v1/todo.md b/docs/v1/todo.md index 3f8f377..7398d99 100644 --- a/docs/v1/todo.md +++ b/docs/v1/todo.md @@ -1,7 +1,8 @@ # JetQueue - version 1 check list -- [ ] Add v1 docs +- [x] Add v1 docs - [ ] Add docs for each package - [ ] Update each package.json files - [ ] Fix web dashboard build time error -- [ ] Polish the codebase for the v1 \ No newline at end of file +- [ ] Polish the codebase for the v1 +- [x] Update deps \ No newline at end of file diff --git a/packages/core/package.json b/packages/core/package.json index 5f424d2..bbbaa16 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -18,10 +18,11 @@ }, "devDependencies": { "@types/bun": "^1.3.13", + "@types/node": "^26.0.1", "tsup": "^8.0.0", "typescript": "^5.3.0" }, - "engines": { + "engines": { "node": ">=22.5.0" }, "peerDependencies": { diff --git a/packages/core/tsconfig.json b/packages/core/tsconfig.json index a9b50ad..8ecddb1 100644 --- a/packages/core/tsconfig.json +++ b/packages/core/tsconfig.json @@ -2,8 +2,7 @@ "extends": "../../tsconfig.json", "compilerOptions": { "outDir": "./dist", - "types": ["bun"], - "baseUrl": ".", + "types": ["bun", "node"], "rootDir": ".", "paths": { "@/*": ["./src/*"] diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a105123..9f4044e 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -126,6 +126,9 @@ importers: '@types/bun': specifier: ^1.3.13 version: 1.3.13 + '@types/node': + specifier: ^26.0.1 + version: 26.0.1 tsup: specifier: ^8.0.0 version: 8.5.1(jiti@2.7.0)(postcss@8.5.14)(typescript@5.9.3) @@ -1161,6 +1164,9 @@ packages: '@types/node@25.6.2': resolution: {integrity: sha512-sokuT28dxf9JT5Kady1fsXOvI4HVpjZa95NKT5y9PNTIrs2AsobR4GFAA90ZG8M+nxVRLysCXsVj6eGC7Vbrlw==} + '@types/node@26.0.1': + resolution: {integrity: sha512-fc3KiUoBt6kie0N9bIW3E47vZsuaMf0PM2AaUpLCLT0s/LvX1nxAim6Fc049cNxODPpGm6qRAuUOB86SkRuPQw==} + '@types/react-dom@19.2.3': resolution: {integrity: sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==} peerDependencies: @@ -2951,6 +2957,9 @@ packages: undici-types@7.19.2: resolution: {integrity: sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg==} + undici-types@8.3.0: + resolution: {integrity: sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ==} + unrs-resolver@1.11.1: resolution: {integrity: sha512-bSjt9pjaEBnNiGgc9rUiHGKv5l4/TGzDmYw3RhnkJGtLhbnnA/5qJj7x3dNDCRx/PJxu774LlH8lCOlB4hEfKg==} @@ -3750,6 +3759,10 @@ snapshots: dependencies: undici-types: 7.19.2 + '@types/node@26.0.1': + dependencies: + undici-types: 8.3.0 + '@types/react-dom@19.2.3(@types/react@19.2.14)': dependencies: '@types/react': 19.2.14 @@ -4415,7 +4428,7 @@ snapshots: eslint: 9.39.4(jiti@2.7.0) eslint-import-resolver-node: 0.3.10 eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint@9.39.4(jiti@2.7.0)))(eslint@9.39.4(jiti@2.7.0)) - eslint-plugin-import: 2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint@9.39.4(jiti@2.7.0)))(eslint@9.39.4(jiti@2.7.0)))(eslint@9.39.4(jiti@2.7.0)) + eslint-plugin-import: 2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.7.0)) eslint-plugin-jsx-a11y: 6.10.2(eslint@9.39.4(jiti@2.7.0)) eslint-plugin-react: 7.37.5(eslint@9.39.4(jiti@2.7.0)) eslint-plugin-react-hooks: 7.1.1(eslint@9.39.4(jiti@2.7.0)) @@ -4468,7 +4481,7 @@ snapshots: tinyglobby: 0.2.16 unrs-resolver: 1.11.1 optionalDependencies: - eslint-plugin-import: 2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint@9.39.4(jiti@2.7.0)))(eslint@9.39.4(jiti@2.7.0)))(eslint@9.39.4(jiti@2.7.0)) + eslint-plugin-import: 2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.7.0)) transitivePeerDependencies: - supports-color @@ -4508,7 +4521,7 @@ snapshots: transitivePeerDependencies: - supports-color - eslint-plugin-import@2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint@9.39.4(jiti@2.7.0)))(eslint@9.39.4(jiti@2.7.0)))(eslint@9.39.4(jiti@2.7.0)): + eslint-plugin-import@2.32.0(@typescript-eslint/parser@8.59.3(eslint@9.39.4(jiti@2.7.0))(typescript@5.9.3))(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.7.0)): dependencies: '@rtsao/scc': 1.1.0 array-includes: 3.1.9 @@ -5892,6 +5905,8 @@ snapshots: undici-types@7.19.2: {} + undici-types@8.3.0: {} + unrs-resolver@1.11.1: dependencies: napi-postinstall: 0.3.4