Skip to content

makefile-pipeline-quick-reference.md

Latest commit

 

History

History
416 lines (335 loc) · 8.94 KB

makefile-pipeline-quick-reference.md

File metadata and controls

416 lines (335 loc) · 8.94 KB
Raw

Makefile CI/CD Pipeline - Quick Reference Guide

Quick Start: make help shows all available commands

Common Workflows

Local Development Deployment

# Complete dev deployment with custom version
make pipeline-dev VERSION=v$(date +%s)

# Or use the workflow integration
make bump

Testing Individual Stages

# Check prerequisites
make check-prerequisites

# Build images only
make build-all-images VERSION=v99999

# Build and push images
make build-all-images push-all-images VERSION=v99999

# Generate and publish Helm chart
make helm-release VERSION=v99999

# Run security scans
make security-scan VERSION=v99999

# Verify deployment health
make verify-deployment-dev

Troubleshooting

Check if prerequisites are installed

make check-prerequisites

Checks: docker, kubectl, helm, envsubst, git

Verify kubeconfig exists

make check-kubeconfig-dev
make check-kubeconfig-stg
make check-kubeconfig-prd

See what images would be built

# Builds but doesn't push (no network access needed except for base images)
make build-all-images VERSION=test
docker images | grep {{PROJECT_NAME}}

Test Helm chart generation

make helm-generate VERSION=test
ls -la helm/server/Chart.yaml helm/server/values.yaml
cat helm/server/Chart.yaml

Pipeline Stages Explained

1. Prerequisites Check

make check-prerequisites
  • Verifies required tools installed
  • Checks: docker, kubectl, helm, envsubst, git
  • Exit code 0: All tools found
  • Exit code 1: Missing tools (shows which ones)

2. Build Images

make build-all-images VERSION=v123

Builds:

  • API server (api:v123)
  • Migration service (migrations:v123)
  • Org provisioner (org-provisioner:v123)
  • Agent ingestion (agent-ingestion:v123)
  • Health controller (health-controller:v123)
  • Probe controller (probe-controller:v123)
  • Customer probe agent binaries (linux-amd64, linux-arm64)

Tags: Both VERSION and latest

3. Push Images

make push-all-images VERSION=v123
  • Pushes all images to Harbor registry
  • Pushes both version-specific and latest tags
  • Requires: Docker login to Docker Hub

4. Helm Chart Release

make helm-release VERSION=v123

Steps:

  1. Generate Chart.yaml and values.yaml from templates
  2. Lint chart
  3. Package as {{PROJECT_NAME}}-server-v123.tgz
  4. Clone/update helm-chart-private repo (SSH)
  5. Copy package and update repo index
  6. Push to GitHub
  7. Poll for chart availability (30 tries, 10 sec intervals)

Requires: SSH key for SupportTools/helm-chart-private

5. Database Migrations

make run-migrations-dev-ci VERSION=v123
make run-migrations-stg-ci VERSION=v123
make run-migrations-prd-ci VERSION=v123
  • Uses pre-built migration image
  • Runs Kubernetes Job
  • Waits for completion (timeout: 10 minutes)
  • Note: These are called by pipeline targets

6. ArgoCD Deployment

make argocd-deploy-dev VERSION=v123
make argocd-deploy-stg VERSION=v123
make argocd-deploy-prd VERSION=v123

Steps:

  1. Apply ArgoCD project
  2. Patch or create ArgoCD Application
  3. Set targetRevision to new version
  4. Wait for Synced + Healthy status (timeout: 7.5 minutes)

7. Deployment Verification

make verify-deployment-dev
make verify-deployment-stg
make verify-deployment-prd

Checks:

  • Pod readiness count
  • Health endpoint (/health)
  • WebSocket connectivity

8. Security Scanning

make security-scan VERSION=v123
  • Installs Trivy if not present
  • Scans all images for CRITICAL+HIGH vulnerabilities
  • Outputs to security-scan-*.txt files
  • Exit code 0: Always succeeds (for CI/CD)

Complete Pipelines

Development Pipeline

make pipeline-dev VERSION=v123

Executes:

  1. check-prerequisites
  2. build-all-images
  3. push-all-images
  4. helm-release
  5. run-migrations-dev-ci
  6. argocd-deploy-dev
  7. verify-deployment-dev

Duration: ~10-15 minutes

Staging Pipeline

make pipeline-stg VERSION=v123

Same as dev, but for staging environment.

Production Pipeline

make pipeline-prd VERSION=v123

Same as dev, but for production environment.

Environment Variables

Required

VERSION                 # Build version (default: timestamp)

Optional (defaults provided)

REGISTRY                # Docker registry (default: docker.io/supporttools)
CHART_REPO_PATH         # Helm chart repo path (default: ../helm-chart-private)
CHART_REPO_URL          # Chart URL (default: https://charts-private.support.tools)

Kubeconfig Paths (auto-configured)

KUBECONFIG_DEV          # Dev cluster config
KUBECONFIG_STG          # Staging cluster config
KUBECONFIG_PRD          # Production cluster config

Common Errors and Solutions

Error: "docker not found"

# Install Docker
sudo apt-get install docker.io

# Or check if it's in PATH
which docker

Error: "envsubst not found"

# Install gettext-base package
sudo apt-get install gettext-base

Error: "Kubeconfig not found"

# Check kubeconfig path
ls -la ~/.kube/mattox/

# Verify path in makefile matches your setup
grep KUBECONFIG makefile

Error: "Chart not available after timeout"

# Check if chart was actually published
cd ../helm-chart-private
git log -1

# Manually verify chart in repository
helm repo update
helm search repo private-charts/{{PROJECT_NAME}}-server --version v123

Error: "Permission denied (publickey)" for helm-chart repo

# Check SSH key is added to ssh-agent
ssh-add -l

# Test SSH connection
ssh -T git@github.com

# Add SSH key if needed
ssh-add ~/.ssh/id_rsa

Error: "ArgoCD deployment timeout"

# Check ArgoCD application status manually
kubectl --kubeconfig $KUBECONFIG_DEV -n argocd get application {{PROJECT_NAME}}-dev

# View ArgoCD application details
kubectl --kubeconfig $KUBECONFIG_DEV -n argocd describe application {{PROJECT_NAME}}-dev

# Check ArgoCD logs
kubectl --kubeconfig $KUBECONFIG_DEV -n argocd logs deployment/argocd-application-controller

Tips and Best Practices

1. Use Specific Versions

# Good - specific version for tracking
make pipeline-dev VERSION=v12345

# Avoid - timestamp makes it hard to track
make pipeline-dev  # Uses timestamp by default

2. Test Locally First

# Always test individual stages before full pipeline
make build-all-images VERSION=test
make helm-generate VERSION=test

3. Check Prerequisites Regularly

# Run after system updates or fresh installs
make check-prerequisites

4. Review Security Scans

# After running security-scan, review results
cat security-scan-api.txt
cat security-scan-migrations.txt

5. Monitor Deployments

# After deployment, check logs
make logs-api-dev

# Check pod status
kubectl --kubeconfig $KUBECONFIG_DEV -n {{PROJECT_NAME}}-dev get pods

# Health check
make health-check-dev

Comparison: Old vs New Workflow

Old Workflow (Shell Scripts)

# API deployment
cd api
./scripts/deploy-dev.sh

# Org management deployment
cd ../controllers/org-management
./scripts/deploy-dev.sh

# Different commands in CI/CD (GitHub Actions inline)

New Workflow (Makefile)

# Same command locally and in CI/CD
make pipeline-dev VERSION=v123

# Or individual components
make build-all-images VERSION=v123
make helm-release VERSION=v123
make argocd-deploy-dev VERSION=v123

Getting Help

Show all available commands

make help

Show workflow commands

make workflow-help

Show GitHub Actions monitoring

make gh-help

Check current task status

make workflow-status

Quick Debugging

What's running in dev?

kubectl --kubeconfig $KUBECONFIG_DEV -n {{PROJECT_NAME}}-dev get pods
kubectl --kubeconfig $KUBECONFIG_DEV -n {{PROJECT_NAME}}-dev get deployments

What version is deployed?

kubectl --kubeconfig $KUBECONFIG_DEV -n {{PROJECT_NAME}}-dev get deployment {{PROJECT_NAME}} -o jsonpath='{.spec.template.spec.containers[0].image}'

Is the API healthy?

make health-check-dev
# Or directly
curl -f https://api-dev.{{PROJECT_NAME}}.com/v1/healthz

What's in the Helm chart repo?

helm repo update
helm search repo private-charts/{{PROJECT_NAME}}-server --versions | head -20

Emergency Procedures

Rollback Deployment

# Patch ArgoCD to previous version
kubectl --kubeconfig $KUBECONFIG_DEV -n argocd patch application {{PROJECT_NAME}}-dev \
  --type merge \
  -p '{"spec":{"source":{"targetRevision":"v12344"}}}'

# Wait for sync
make argocd-wait-dev

Force Pod Restart

make restart-api-dev
# Or manually
kubectl --kubeconfig $KUBECONFIG_DEV -n {{PROJECT_NAME}}-dev rollout restart deployment/{{PROJECT_NAME}}

Check Recent Changes

# GitHub Actions runs
make gh-status

# Git commits
git log --oneline -10

# Helm chart versions
cd ../helm-chart-private && git log --oneline -10