diff --git a/docs/deployment.md b/docs/deployment.md index 644e05a..8d79704 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -3,357 +3,152 @@ !!! tip The easiest way to run OpenCVE is to use [OpenCVE Cloud](https://www.opencve.io/) (no installation, configuration or maintenance needed). -You can install the complete OpenCVE stack with Docker. It will contain all the services you need to run OpenCVE (Django webserver, Airflow services, Redis, PostgreSQL). +You can install the complete OpenCVE stack with Docker. It includes all the services needed to run OpenCVE (Django webserver, Airflow, Redis, PostgreSQL, Nginx). -## Requirements - -You need to have your server installed with **Debian OS** or equivalent, **Docker v25.0.x** at least and the **docker-compose** plugin. - -The procedure is run as root and has been tested with Debian 11 and Debian 12. - -The minimum requirement for the host is **4 cores and 4 GB RAM**, the disk usage will be around 25GB or 30GB. The instance [d2-8](https://www.ovhcloud.com/en/public-cloud/sandbox/) from OVHcloud is a good fit to start. - -## Installation +## Quickstart -Clone the OpenCVE [repository](https://github.com/opencve/opencve) and go into the `docker/` directory. +```bash +git clone https://github.com/opencve/opencve.git +cd opencve/docker -Then run the `install.sh` script: - -``` -~/opencve/docker$ ./install.sh -``` - -!!! note - This command installs the [latest stable release](https://github.com/opencve/opencve/releases). If you want to install the very latest available commit, use `./install.sh -r master`. - For details about the command: `./install.sh -h`. +./install.sh prepare # default: master branch +# edit docker/.env and web/opencve/conf/.env (passwords) -As a first step, it will prepare your environment with the default configuration files: +./install.sh start # ~30-60 min +# OpenCVE β†’ http://localhost:80 +# Airflow β†’ http://localhost:8080 ``` -🟒 The default configuration files are all set, you can update them now if you want before starting the entire docker stack: - Docker compose : ./.env - Webserver : ../web/opencve/conf/.env - Airflow : ../scheduler/airflow.cfg - Django : ../web/opencve/conf/settings.py - -See the documentation for details: https://docs.opencve.io/deployment/#configuration - πŸš€ You can now run: ./install.sh start -``` +See [Configuration](deployment/configuration.md) for the settings to review before `./install.sh start`. +For a specific release: `./install.sh -r v3.0.0 prepare && ./install.sh start` -!!! info - To customize the default settings before starting the OpenCVE docker stack, you can refer to our [configuration section](deployment.md#configuration) of this documentation. +If the Docker build fails due to network issues, see [Troubleshooting](troubleshooting.md#docker-build-fails-due-to-network-issues). +## Requirements -Then you can run the second part of the installation: +- **OS**: Debian 11/12 or equivalent (also tested on similar Linux distributions) +- **Docker**: v25.0.x or later with the **Docker Compose V2** plugin (`docker compose`) +- **Git**: required only to clone the repository (not used by `install.sh` itself) +- **User**: a user with permission to run Docker (root or member of the `docker` group) +- **Hardware**: 4 cores, 4 GB RAM minimum; ~25–30 GB disk space -``` -./install.sh start -``` +The Discovery [d2-8](https://www.ovhcloud.com/en/public-cloud/prices/) instance from OVHcloud is a good fit to start. -It will proceed with all the installation steps automatically. When all the steps are done, the installation will end like this: +## Architecture +```mermaid +flowchart LR + User([Browser]) --> Nginx + Nginx --> Webserver[Django webserver] + Webserver --> Postgres[(PostgreSQL)] + Webserver --> Repos[(repositories volume)] + Scheduler[Airflow scheduler] --> Postgres + Scheduler --> Redis[(Redis)] + Worker[Airflow worker] --> Redis + Worker --> Postgres + Worker --> Repos + User --> Airflow[Airflow webserver] + Airflow --> Scheduler ``` -🟒 Everything is set up, you can now access to OpenCVE locally: - -- on port 80 for OpenCVE web instance - You can login with your account and password set at the previous step 'Create OpenCVE admin user'. - You can set a new one with the following command: ./install.sh create-superuser - The installed version is: v2.0.0 - -- on port 8080 for OpenCVE Airflow scheduler - You can login with the username "xxx" and password "xxx". - - πŸ‘‹ Installation complete! Thank you for choosing OpenCVE! - -``` +The local Git clone provides Docker configuration files and examples. Application code (web + DAGs) is fetched from GitHub and **built into Docker images** using `OPENCVE_VERSION` in `docker/.env`. To update the running application, use `./install.sh upgrade`, not `git pull`. -## List of Commands +## Installation -You can run dedicated commands of the `install.sh` script if you want to proceed step by step once the configuration files are set : +### Step 1 β€” Clone the repository ``` - init-docker-stack - clone-repositories - create-superuser - import-opencve-kb - start-opencve-dag - install-end +git clone https://github.com/opencve/opencve.git +cd opencve/docker ``` -!!! note - For details about each command: `./install -h`. - -### init-docker-stack - -This command is used to bootstrap the whole OpenCVE stack. - -It will create the containers, add the airflow connections, collect the web static files, update the `OPENCVE_SECRET_KEY` and create the needed database tables and procedures: +### Step 2 β€” Prepare configuration files ``` -./install.sh init-docker-stack - ---------| Docker compose up -> starting OpenCVE docker stack... -[...] - βœ” Network opencve_default Created 0.2s - βœ” Volume "opencve_staticfiles" Created 0.0s - βœ” Volume "opencve_postgres-db" Created 0.0s - βœ” Volume "opencve_repositories" Created 0.0s - βœ” Container webserver Started 1.1s - βœ” Container redis Healthy 6.6s - βœ” Container nginx Started 1.2s - βœ” Container postgres Healthy 6.6s - βœ” Container airflow-init Exited 22.5s - βœ” Container airflow-webserver Started 22.9s - βœ” Container airflow-scheduler Started 22.9s - βœ” Container airflow-worker Started 22.9s -> done βœ… - ---------| Collect static files from Django webserver -> collecting latest static files... -> done βœ… - ---------| Apply Django webserver DB migration -> migrating DB schema with latest changes... -> done βœ… - ---------| Airflow connections -> adding Postgresql Airflow connection... -> done βœ… -> adding Redis Airflow connection... -> done βœ… - ---------| OpenCVE secret key -> cleaning old OpenCVE secret key... -> done βœ… -> updating with new OpenCVE secret key... -> done βœ… - ---------| Webserver restart -> restarting webserver docker instance... -> done βœ… +./install.sh prepare ``` -You can check if the docker instances are running correctly: - -``` -docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -b6c6ff28f06d opencve-airflow-worker "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp airflow-worker -7598b5df72f6 opencve-airflow-webserver "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp airflow-webserver -004246bc3ff6 opencve-airflow-scheduler "/usr/bin/dumb-init …" About a minute ago Up About a minute (healthy) 8080/tcp airflow-scheduler -e4c55d1022ac nginx:bookworm "/docker-entrypoint.…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp nginx -50a2d2f33f90 postgres:15 "docker-entrypoint.s…" About a minute ago Up About a minute (healthy) 5432/tcp postgres -ec7671ac36b3 opencve-webserver "gunicorn opencve.ws…" About a minute ago Up About a minute 8000/tcp webserver -3b32b8df0a9a redis/redis-stack:latest "/entrypoint.sh" About a minute ago Up About a minute (healthy) 6379/tcp, 8001/tcp redis -``` +Running `./install.sh` with no arguments is equivalent to `./install.sh prepare`. -### clone-repositories +!!! note + Default branch is **master**. Application code is built into Docker images (see `OPENCVE_VERSION` in `docker/.env`). + For a specific [release](https://github.com/opencve/opencve/releases): `./install.sh -r v3.0.0 prepare`. + Full option list: [Reference](deployment/install.md). -This command is used to clone the repositories needed for OpenCVE to work: +Expected output: ``` -./install.sh clone-repositories - ---------| Setup OpenCVE repositories -> cloning opencve-kb... -Cloning into '/home/airflow/repositories/opencve-kb'... -Updating files: 100% (277971/277971), done. -> done βœ… -> cloning opencve-nvd... -Cloning into '/home/airflow/repositories/opencve-nvd'... -Updating files: 100% (277955/277955), done. -> done βœ… -> cloning opencve-redhat... -Cloning into '/home/airflow/repositories/opencve-redhat'... -Updating files: 100% (35928/35928), done. -> done βœ… -> cloning cvelistV5... -Cloning into '/home/airflow/repositories/cvelistV5'... -Updating files: 100% (278007/278007), done. -> done βœ… -> cloning vulnrichment... -Cloning into '/home/airflow/repositories/vulnrichment'... -Updating files: 100% (61734/61734), done. -> done βœ… -``` - -### import-opencve-kb +🟒 The configuration files are all set, you can update them now if you want before starting the entire docker stack: + Nginx : ./conf/opencve.conf.template + Airflow : ../scheduler/airflow.cfg + Webserver : ../web/opencve/conf/.env & ../web/opencve/conf/settings.py + Docker compose : ./.env -This command imports the [OpenCVE KB](concepts/kb.md) in your database: +See the documentation for details: https://docs.opencve.io/deployment/configuration/ -``` -./install.sh import-opencve-kb - ---------| Import OpenCVE KB inside the database, this can take 10 to 30min. -> importing CVEs... -Parsing the OpenCVE KB repository (/app/repositories/opencve-kb) -Found 277970 CVEs, adding them in database... -Done in 659.365s -> done βœ… + πŸš€ You can now run: ./install.sh start ``` -### start-opencve-dag +### Step 3 β€” Configure (before first start) -This command unpauses the main DAG so Airflow can execute the `opencve` workflow: - -``` -./install.sh start-opencve-dag +!!! warning + Change default passwords and secrets in `docker/.env` **before** running `./install.sh start`. If you change `POSTGRES_PASSWORD`, update `OPENCVE_DATABASE_URL` in `web/opencve/conf/.env` as well. ---------| Unpause Airflow OpenCVE dag to start to update local repositories and alerts -> unpausing OpenCVE Airflow dag... -> done βœ… -``` +Minimum checklist: -!!! note - You may have the following warning: - `/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/dot_renderer.py:28 UserWarning: Could not import graphviz. Rendering graph to the graphical format will not be possible.` - It doesn't impact OpenCVE as this library is not used. +- `POSTGRES_PASSWORD` in `docker/.env` +- `_AIRFLOW_WWW_USER_USERNAME` / `_AIRFLOW_WWW_USER_PASSWORD` in `docker/.env` +- `AIRFLOW__CORE__FERNET_KEY` in `docker/.env` +- `OPENCVE_DATABASE_URL` in `web/opencve/conf/.env` (if PostgreSQL credentials changed) -### create-superuser +See the full [Configuration](deployment/configuration.md) guide for all settings, ports, and Nginx templates. -This command creates the superuser needed to access to the admin part of your OpenCVE website: +### Step 4 β€” Start the stack ``` -$ ./install.sh create-superuser - ---------| Create admin user on OpenCVE -> creating OpenCVE admin user... -Username: xxx -Email address: xxx -Password: -Password (again): -Superuser created successfully. -> done βœ… - ---------| Auto confirm the created user -> confirming the created admin user... -> done βœ… +./install.sh start ``` -## Configuration - -**docker/.env** - -We recommend to verify and update your docker `.env` file. It's advised to change the following variables: - -- `POSTGRES_PASSWORD` -- `_AIRFLOW_WWW_USER_USERNAME` -- `_AIRFLOW_WWW_USER_PASSWORD` -- `AIRFLOW__CORE__FERNET_KEY` - -!!! warning - You need to set the above env variables before starting OpenCVE stack as they are configured at the first run of docker compose. - -To customize the fernet key: [how to generate a new fernet key](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/fernet.html#generating-fernet-key). - -**web/opencve/conf/.env** - -This file exposes important settings used by the OpenCVE webserver: - -- `OPENCVE_DATABASE_URL`: the URI of the database. You need to update it if you have changed some of the postgresql parameters in the docker compose env file. The postgresql connection parameters need to be the same in both web and docker compose `.env` files. -- `OPENCVE_SECRET_KEY`: this variable has to be replaced with a new secret when the web server is up the first time -- `OPENCVE_*_REPO_PATH`: the paths of the repositories (KB, Mitre, NVD...) -- `OPENCVE_V1_DATABASE`: the URI of an OpenCVE v1 instance (used for [data migration](guides/migrate_opencve_v1.md)) - - -**web/opencve/conf/settings.py** - -This file is used to override the Django default settings defined in [web/opencve/conf/base.py](https://github.com/opencve/opencve/blob/v2/web/opencve/conf/base.py). - -You can check the list of Django settings [here](https://docs.djangoproject.com/en/5.1/ref/settings/) to customize your OpenCVE instance. - -**scheduler/airflow.cfg** - -This file is used to configure the Airflow scheduler of OpenCVE. - -You can check the list of Airflow configurations [here](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html) to customize your Airflow instance. - -!!! info - You can refer to this [dedicated guide](guides/smtp_configuration.md) to configure SMTP settings and enable email sending. - -## Upgrade - -Upgrading OpenCVE involves updating the Docker containers and running specific maintenance commands to ensure everything functions correctly. - -!!! warning "Important" - Before proceeding with the upgrade, we strongly recommend making a backup of your database to prevent any potential data loss. Additionally, review the [Releases page](https://github.com/opencve/opencve/releases) for specific details and instructions about each new version. - -Access the Airflow webserver and pause the opencve DAG to prevent it from running during the upgrade (ensure no DAG runs are in progress before pausing): - -![Pause DAG](images/deployment/pause-dag.png){.center style="width:75%"} - -!!! tip - You can also use the following commands to pause / unpause the DAG: +This runs all installation steps automatically (about 30–60 minutes): - ``` - docker compose exec airflow-scheduler airflow dags pause opencve - docker compose exec airflow-scheduler airflow dags unpause opencve - ``` +1. `init-docker-stack` β€” build & start containers, migrate DB, collect static files, Airflow connections, secret key +2. `clone-repositories` β€” clone KB repositories into the Docker volume +3. `verify-container-versions` β€” ensure images match `OPENCVE_VERSION` +4. `start-opencve-dag` β€” unpause all Airflow DAGs +5. `create-superuser` β€” interactive admin account creation +6. `install-end` β€” display access URLs and credentials summary -Navigate to the OpenCVE project directory and pull the latest updates: +When complete: ``` -cd opencve && git pull -``` - -Then navigate to the docker folder and update the `OPENCVE_VERSION` variable of the `.env`: -``` -cd docker -~/opencve/docker$ grep OPENCVE_VERSION .env -OPENCVE_VERSION=v2.0.0 -``` +🟒 Everything is set up, you can now access to OpenCVE locally: -!!! note - You can select the desired version in the [releases list](https://github.com/opencve/opencve/releases), or you can use `master` to install the very latest available commit of OpenCVE. +- on port 80 for OpenCVE web instance + You can login with your account and password set at the previous step 'Create OpenCVE admin user'. + You can set a new one with the following command: ./install.sh create-superuser + The installed version is: master -Update the Docker containers to reflect the latest code and dependencies, using `install.sh`: +- on port 8080 for OpenCVE Airflow scheduler + You can login with the username "xxx" and password "xxx". -``` -~/opencve/docker$ ./install.sh docker-build + πŸ‘‹ Installation complete! Thank you for choosing OpenCVE! ---------| Docker compose build -> building OpenCVE docker images... -[...] -> done βœ… ``` -After the container images are built, execute the `docker-up` command that will up the containers and with Django, it will collect static files and apply any database migrations: +Ports depend on `OPENCVE_PORT` and `AIRFLOW_WEBSERVER_PORT` in `docker/.env`. -``` -~/opencve/docker$ ./install.sh docker-up - ---------| Docker compose up -> starting OpenCVE docker stack... -[+] Running 8/8 - βœ” Container redis Healthy 3.9s - βœ” Container nginx Running 0.0s - βœ” Container webserver Started 3.7s - βœ” Container postgres Healthy 3.9s - βœ” Container airflow-init Exited 22.0s - βœ” Container airflow-worker Started 22.5s - βœ” Container airflow-webserver Started 22.5s - βœ” Container airflow-scheduler Started 22.4s -> done βœ… - ---------| Collect static files from Django webserver -> collecting latest static files... -> done βœ… - ---------| Apply Django webserver DB migration -> migrating DB schema with latest changes... -> done βœ… -``` +!!! info + Configure email notifications after installation with the [SMTP guide](guides/smtp_configuration.md). -You can finally [unpause](deployment.md#start-opencve-dag) the `opencve` DAG to to restart the periodic tasks and resume normal operations. +See the [Reference](deployment/install.md) for all script options and commands. -!!! note - Since OpenCVE relies on Django and Airflow, it is good practice to review their upgrade documentation for additional guidance: +## Further reading - - [Django Upgrade Guide](https://docs.djangoproject.com/en/5.1/howto/upgrade-version/) - - [Airflow Upgrade Guide](https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading.html) +- [Configuration](deployment/configuration.md) β€” all config files and variables +- [Operations](deployment/operations.md) β€” upgrade, reset, and backup +- [Reference](deployment/install.md) β€” `install.sh` options and commands +- [Troubleshooting](troubleshooting.md) β€” common issues diff --git a/docs/deployment/configuration.md b/docs/deployment/configuration.md new file mode 100644 index 0000000..c92057b --- /dev/null +++ b/docs/deployment/configuration.md @@ -0,0 +1,79 @@ +# Configuration + +This page describes all configuration files used by the OpenCVE Docker stack. Review them **before** running `./install.sh start` for the first time. + +The `prepare` command (or `./install.sh` with no arguments) copies example files to their active locations: + +| File | Purpose | +|------|---------| +| `docker/.env` | Docker Compose environment variables | +| `docker/conf/opencve.conf.template` | Nginx configuration for OpenCVE | +| `docker/conf/default.conf.template` | Nginx default site template | +| `web/opencve/conf/.env` | Django webserver environment | +| `web/opencve/conf/settings.py` | Django settings overrides | +| `scheduler/airflow.cfg` | Airflow scheduler configuration | + +Use `./install.sh -f prepare` to replace existing configuration files with fresh defaults from the examples. + +## docker/.env + +We recommend verifying and updating your docker `.env` file before the first start. + +### Security (change before first start) + +- `POSTGRES_PASSWORD` +- `_AIRFLOW_WWW_USER_USERNAME` +- `_AIRFLOW_WWW_USER_PASSWORD` +- `AIRFLOW__CORE__FERNET_KEY` + +!!! warning + These variables are applied when Docker Compose creates the containers for the first time. Change them **before** running `./install.sh start`. + If you change `POSTGRES_PASSWORD`, update `OPENCVE_DATABASE_URL` in `web/opencve/conf/.env` accordingly before starting the stack. + +!!! note + To customize the fernet key: [how to generate a new fernet key](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/fernet.html#generating-fernet-key). + +### Version and ports + +- `OPENCVE_REPOSITORY`: Git repository URL (default: `https://github.com/opencve/opencve.git`) +- `OPENCVE_VERSION`: release tag or branch built into Docker images (default: `master`, set by `./install.sh prepare`) +- `OPENCVE_PORT`: port exposed for the OpenCVE web interface (default: `80`) +- `AIRFLOW_WEBSERVER_PORT`: port exposed for the Airflow UI (default: `8080`) + +The application code is fetched from `OPENCVE_REPOSITORY` at `OPENCVE_VERSION` during the Docker build. Updating the running application does **not** require a `git pull` of the application code, use `./install.sh upgrade` instead (see [Operations](operations.md#upgrade)). + +## web/opencve/conf/.env + +This file exposes important settings used by the OpenCVE webserver: + +- `OPENCVE_DATABASE_URL`: database URI. Update it if you changed PostgreSQL parameters in `docker/.env`. Connection parameters must match in both files. +- `OPENCVE_SECRET_KEY`: generated automatically by `./install.sh` on first run (skipped if already set) +- `OPENCVE_*_REPO_PATH`: paths to the KB repositories (KB, MITRE, NVD, etc.) +- `OPENCVE_V1_DATABASE`: URI of an OpenCVE v1 instance if needed (used for [data migration](../guides/migrate_opencve_v1.md)) + +## web/opencve/conf/settings.py + +This file overrides Django default settings defined in [web/opencve/conf/base.py](https://github.com/opencve/opencve/blob/master/web/opencve/conf/base.py). + +See the [Django settings reference](https://docs.djangoproject.com/en/5.1/ref/settings/) to customize your instance. + +## scheduler/airflow.cfg + +This file configures the Airflow scheduler of OpenCVE. + +See the [Airflow configuration reference](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html) to customize your Airflow instance. + +The `prepare` command sets the DAG start dates to the current date. + +## Nginx templates + +Nginx configuration is managed through templates in `docker/conf/`: + +- `opencve.conf.template`: routes traffic to the Django webserver and serves static files +- `default.conf.template`: default site configuration + +These are processed by the official `nginx:trixie` image at container startup. + +## Email (SMTP) + +To enable email notifications, configure SMTP settings in `scheduler/airflow.cfg` and trigger the `check_smtp` DAG. See the dedicated [SMTP configuration guide](../guides/smtp_configuration.md). diff --git a/docs/deployment/install.md b/docs/deployment/install.md new file mode 100644 index 0000000..8ce9ca9 --- /dev/null +++ b/docs/deployment/install.md @@ -0,0 +1,157 @@ +# Reference + +This page documents the `install.sh` script: its options and commands. + +For a guided first install, see [Overview](../deployment.md#installation). + +## Options + +| Option | Description | Example | +|--------|-------------|---------| +| `-h` | Print usage | `./install.sh -h` | +| `-r` | Release or branch to install (default: `master`) | `./install.sh -r v3.0.0 prepare` | +| `-f` | Force overwrite of existing configuration files | `./install.sh -f prepare` | +| `-y` | Skip confirmation prompts (`reset` only) | `./install.sh -y reset` | + +## Commands + +| Command | When to use | +|---------|-------------| +| `prepare` | First install, or refresh config files | +| `start` | First install (after prepare + configuration) | +| `upgrade` | Update to a new version β€” see [Operations](operations.md#upgrade) | +| `reset` | Destroy all data and reinstall β€” see [Operations](operations.md#reset) | +| `create-superuser` | Create or reset the admin account | +| `docker-build` | Rebuild images manually | +| `docker-up` | Start containers, migrate, collect static files | +| `init-docker-stack` | Bootstrap containers and connections | +| `clone-repositories` | Clone KB repos into the Docker volume | +| `import-opencve-kb` | Import CVEs into PostgreSQL (one-time) | +| `start-opencve-dag` | Unpause all Airflow DAGs | +| `init-secret-key` | Generate Django secret key | +| `bye` | Display installation summary (URLs, version) | + +Most commands are **idempotent**: they skip steps already completed (existing repositories, imported CVEs, superuser, secret key, etc.). + +For step-by-step installation, run individual commands from the table above instead of `start`. + +!!! note + Full command details: `./install.sh -h`. + +## init-docker-stack + +Bootstraps the whole OpenCVE stack: builds and starts containers, applies migrations, collects static files, configures Airflow connections, and generates `OPENCVE_SECRET_KEY` if not already set. + +``` +./install.sh init-docker-stack + +--------| Docker compose up (1/5) +> starting OpenCVE docker stack... +[...] +> done βœ… + +--------| Apply Django webserver DB migration (2/5) +> migrating DB schema with latest changes... +> done βœ… + +--------| Collect static files from Django webserver (3/5) +> collecting latest static files... +> done βœ… + +--------| Add Airflow connections (4/5) +> adding opencve_postgres Airflow connection... +> done βœ… +> adding opencve_redis Airflow connection... +> done βœ… + +--------| Generate OpenCVE secret key (5/5) +> updating OpenCVE secret key... +> done βœ… +``` + +Verify running containers: + +``` +docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +9fb104f950f9 opencve-airflow-webserver "/usr/bin/dumb-init …" 1 hour ago Up 1 hour (healthy) 0.0.0.0:8080->8080/tcp opencve-airflow-webserver-1 +00f090d9a66c opencve-airflow-scheduler "/usr/bin/dumb-init …" 1 hour ago Up 1 hour (healthy) 8080/tcp opencve-airflow-scheduler-1 +8907176332df opencve-airflow-worker "/usr/bin/dumb-init …" 1 hour ago Up 1 hour (healthy) 8080/tcp opencve-airflow-worker-1 +d9bea2556edc postgres:15 "docker-entrypoint.s…" 1 hour ago Up 1 hour (healthy) 5432/tcp opencve-postgres-1 +672d2db6326d nginx:trixie "/docker-entrypoint.…" 1 hour ago Up 1 hour 0.0.0.0:80->80/tcp opencve-nginx-1 +da5a51bdb679 redis/redis-stack:latest "/entrypoint.sh" 1 hour ago Up 1 hour (healthy) 6379/tcp, 8001/tcp opencve-redis-1 +e762d16ce6f2 opencve-webserver "gunicorn opencve.ws…" 1 hour ago Up 1 hour 8000/tcp opencve-webserver-1 +``` + +## clone-repositories + +Clones the KB repositories into the `repositories` Docker volume. Skips repositories that are already present (ongoing updates are handled by the OpenCVE DAG). + +``` +./install.sh clone-repositories + +--------| Initialize OpenCVE repositories (1/1) +> cloning opencve-kb... +> done βœ… +> cloning opencve-nvd... +> done βœ… +[...] +``` + +## import-opencve-kb + +Imports the [OpenCVE KB](../concepts/kb.md) into PostgreSQL. Skipped if CVEs are already present. Expect **10 to 30 minutes** depending on hardware. + +``` +./install.sh import-opencve-kb + +--------| Import OpenCVE KB inside the database, it can take 10 to 30 min (1/1) +> importing CVEs... +> done βœ… +``` + +## start-opencve-dag + +Unpauses all OpenCVE DAGs: `opencve`, `summarize_reports`, `sync_weaknesses`, `clean_reports`. + +``` +./install.sh start-opencve-dag + +--------| Unpause Airflow DAGs (1/1) +> unpausing opencve dag... +> done βœ… +> unpausing summarize_reports dag... +> done βœ… +> unpausing sync_weaknesses dag... +> done βœ… +> unpausing clean_reports dag... +> done βœ… +``` + +!!! note + You may see a warning about `graphviz` not being installed. It does not impact OpenCVE. + +## create-superuser + +Creates the admin account for OpenCVE. Skipped if a superuser already exists. + +``` +$ ./install.sh create-superuser + +--------| Create OpenCVE admin user (1/2) +> creating OpenCVE admin user... +Username: xxx +Email address: xxx +Password: +Password (again): +Superuser created successfully. +> done βœ… + +--------| Auto confirm the created user (2/2) +> confirming the created admin user... +> done βœ… +``` + +## install-end / bye + +Displays a summary of access URLs, installed version, and credentials. Run manually with `./install.sh bye`, or it runs automatically at the end of `start` and `upgrade`. diff --git a/docs/deployment/operations.md b/docs/deployment/operations.md new file mode 100644 index 0000000..296e98a --- /dev/null +++ b/docs/deployment/operations.md @@ -0,0 +1,91 @@ +# Operations + +This page covers day-to-day operations: upgrading, resetting, and backing up your OpenCVE instance. + +## Backup + +Before any upgrade or destructive operation, back up your PostgreSQL database: + +``` +~/opencve/docker$ docker compose exec -T postgres pg_dump -U opencve opencve | gzip > opencve-backup-$(date +%Y%m%d).sql.gz +``` + +Adjust the username if you changed `POSTGRES_USER` in `docker/.env`. + +To restore from a backup: + +``` +~/opencve/docker$ gunzip -c opencve-backup-YYYYMMDD.sql.gz | docker compose exec -T postgres psql -U opencve opencve +``` + +## Upgrade + +Upgrading OpenCVE rebuilds the Docker images at the target version and runs database migrations. + +!!! warning + Back up your database before upgrading. Review the [Releases page](https://github.com/opencve/opencve/releases) for version-specific notes. + +Navigate to the `docker/` directory and run: + +``` +~/opencve/docker$ ./install.sh upgrade +``` + +This installs the latest `master` branch by default. The script will: + +1. Update `OPENCVE_VERSION` in `docker/.env` +2. Pause all Airflow DAGs and wait for running tasks to finish (up to 5 minutes) +3. Rebuild and restart the webserver, then apply migrations and collect static files +4. Rebuild and restart the Airflow stack and nginx +5. Verify that container versions match `OPENCVE_VERSION` +6. Reconfigure Airflow connections and unpause all DAGs + +To upgrade to a specific release: + +``` +~/opencve/docker$ ./install.sh -r v3.0.0 upgrade +``` + +!!! tip + You can also pause / unpause DAGs manually: + + ``` + docker compose exec airflow-scheduler airflow dags pause opencve + docker compose exec airflow-scheduler airflow dags unpause opencve + ``` + +!!! note + The application code is pulled from GitHub at build time via `OPENCVE_VERSION`. A `git pull` on your local clone only updates Docker configuration files and `install.sh` β€” it does not update the running application. + + Since OpenCVE relies on Django and Airflow, review their upgrade documentation for additional guidance: + + - [Django Upgrade Guide](https://docs.djangoproject.com/en/5.1/howto/upgrade-version/) + - [Airflow Upgrade Guide](https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading.html) + +## Reset + +The `reset` command stops the stack and removes all Docker volumes (database, repositories, static files). Configuration files on the host are kept. + +``` +~/opencve/docker$ ./install.sh reset +``` + +The script asks for confirmation (`Type 'yes' to confirm destruction`). Use `-y` to skip the prompt: + +``` +~/opencve/docker$ ./install.sh -y reset +``` + +After a reset, reinstall with: + +``` +./install.sh prepare && ./install.sh start +``` + +The KB import step alone can take 10 to 30 minutes on a fresh install. + +To also replace configuration files with fresh defaults: + +``` +./install.sh reset && ./install.sh -f prepare && ./install.sh start +``` diff --git a/docs/js/redirects.js b/docs/js/redirects.js new file mode 100644 index 0000000..9d239a1 --- /dev/null +++ b/docs/js/redirects.js @@ -0,0 +1,25 @@ +(function () { + var deploymentAnchors = { + upgrade: "operations/#upgrade", + configuration: "configuration/", + "list-of-commands": "install/", + "init-docker-stack": "install/#init-docker-stack", + "clone-repositories": "install/#clone-repositories", + "import-opencve-kb": "install/#import-opencve-kb", + "start-opencve-dag": "install/#start-opencve-dag", + "create-superuser": "install/#create-superuser", + }; + + if (!window.location.pathname.match(/\/deployment\/?$/)) { + return; + } + + var hash = window.location.hash.slice(1); + var target = deploymentAnchors[hash]; + if (!target) { + return; + } + + var base = window.location.pathname.replace(/\/?$/, "/"); + window.location.replace(base + target); +})(); diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 446e00d..a42f776 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -53,7 +53,7 @@ E: Unable to locate package git This means the container could not access the internet to fetch packages. This issue is related to your **host machine or Docker network configuration**, not to OpenCVE itself. -You can work around the issue by forcing Docker to use the host network during the build. Edit your `docker-compose.yml` and add `network: host` under the `build` section of the service that fails, for example: +You can work around the issue by forcing Docker to use the host network during the build. Edit your `docker-compose.yaml` and add `network: host` under the `build` section of the service that fails, for example: ``` webserver: @@ -85,7 +85,7 @@ Because this behavior was undocumented, OpenCVE’s scheduler failed to parse th The problem has been fixed in this [pull request](https://github.com/opencve/opencve/pull/604). -You can upgrade OpenCVE and relaunch the failed DAG run in your Airflow scheduler instance. To do so, select it: +You can upgrade OpenCVE with `./install.sh upgrade` (see [Operations β†’ Upgrade](deployment/operations.md#upgrade)) and relaunch the failed DAG run in your Airflow scheduler instance. To do so, select it: ![Troubleshooting Select DAGRun](images/troubleshooting/select-dagrun.png){.center style="width:100%"} diff --git a/mkdocs.yml b/mkdocs.yml index cd0705c..ef8addc 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,14 +18,25 @@ plugins: extra_css: - css/extra.css +extra_javascript: + - js/redirects.js + markdown_extensions: - admonition - attr_list - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format nav: - Home: 'index.md' - - Deployment & Setup: 'deployment.md' + - Deployment & Setup: + - Overview: 'deployment.md' + - Configuration: 'deployment/configuration.md' + - Operations: 'deployment/operations.md' + - Reference: 'deployment/install.md' - Concepts: - Core Concepts: 'concepts/core.md' - Knowledge Base (KB): 'concepts/kb.md'