diff --git a/subteams/simulation/flying.md b/subteams/simulation/flying.md index 25eccf9..d520557 100644 --- a/subteams/simulation/flying.md +++ b/subteams/simulation/flying.md @@ -6,53 +6,194 @@ permalink: /simulation/flying/ [Back to Simulation Docs](/docs/simulation/) -This page assumes you've followed the steps in [Environment Setup (Windows)](/docs/simulation/install/windows) or [Environment Setup (Linux)](/docs/simulation/install/linux). +This page assumes you have completed [Environment Setup (Windows)](/docs/simulation/install/windows) and can start the containers as described on the [container setup page](/docs/simulation/containers/). ## Table of Contents -- [Flying the Virtual Drone with Code](#flying-the-virtual-drone-with-code) - - [Table of Contents](#table-of-contents) - - [During Development (Unreal Editor and running code manually)](#during-development-unreal-editor-and-running-code-manually) - - [Configuring the Environment](#configuring-the-environment) - - [Single-Drone](#single-drone) - - [Multi-Drone Settings](#multi-drone-settings) - - [More on `.\update_airsim_settings.ps1`](#more-on-update_airsim_settingsps1) - - [Flying](#flying) - - [Production](#production) +- [How the pieces fit together](#how-the-pieces-fit-together) +- [Start order matters](#start-order-matters) +- [Running the interface script](#running-the-interface-script) +- [Ports](#ports) +- [Flying more than one drone](#flying-more-than-one-drone) +- [Shutting down](#shutting-down) +- [Troubleshooting](#troubleshooting) +- [Production](#production) +- [Legacy: configuring AirSim with settings.json](#legacy-configuring-airsim-with-settingsjson) -## During Development (Unreal Editor and running code manually) +## How the pieces fit together -### Configuring the Environment +Three programs run at once, split across two systems: -Ensure that your AirSim settings are up to date using the `./update_airsim_settings.ps1` command located in the root of the `SUAS` repository. The essence of this script is copying a file to where AirSim expects a `settings.json` file. We have a few pre-made templates available, but you can refer to the [Project AirSim Settings Documenation](https://github.com/iamaisim/ProjectAirSim/blob/main/docs/config.md) or [AirSim Settings Documentation](https://microsoft.github.io/AirSim/settings/) for more details. +| Where | What it does | +| --- | --- | +| Windows | Unreal Engine (Project AirSim as a plugin) includes: the scene, the physics, and the simulated sensors | +| WSL, `env` container | The flight code, plus the interface script that stands the scene up and drives it | +| WSL, `sim` container | ArduPilot SITL — the actual autopilot firmware, one instance per drone | -#### Single-Drone +Nothing here runs Unreal inside a container. Unreal runs natively on Windows, and the containers talk to it over the network. -Flying a single drone is easiest. For a quick start, you can run the following command from the SUAS repository root to set up your settings: +Three connections have to come up, and they depend on each other: + +1. The interface script connects to Unreal over the Project AirSim API. +2. Unreal sends sensor data to the SITL over UDP; the SITL sends servo output back. +3. Your flight code connects to the SITL over MAVLink on TCP port `5762` (and `5772`, `5782`, ... for additional drones — the SITL adds 10 per instance). + +## Start order matters + +**Start these in order.** The interface script loads an *empty* scene first and then waits for you, on purpose: the SITL needs *a* scene to pull data from when it starts, but a drone that spawns before its SITL exists will lock up, and only a full restart recovers it. The prompt in the script is the gap between those two requirements. + +### 1. Unreal + +Open the Project AirSim Unreal project, open the level you want, and press **Play**. + +- The Play button is above the viewport to the far right. If it is not immediately visible, press the double-arrow button, then `Play`. +- The Unreal Editor may appear to freeze once the simulation is running. This is normal. + +This must be running first — the interface script connects to Unreal as its very first action, and will simply fail if Unreal is not in Play mode. + +### 2. The `env` container + +In a WSL terminal: + +```bash +./simulation/run_container.sh env +``` + +This drops you into a shell inside the container, with your competition repository mounted at `/workspace`. From there, start the interface script (see [Running the interface script](#running-the-interface-script) below). + +It will connect to Unreal, load the empty scene, and then stop at a prompt reading: + +``` +Start your sim container now. Press enter to continue (add drones to scene) +``` + +**Do not press Enter yet.** + +### 3. The `sim` container + +Leave the first terminal sitting at that prompt. In a **second** WSL terminal: + +```bash +./simulation/run_container.sh sim +``` + +This starts the ArduPilot SITL. Wait until it reports that it is waiting for a heartbeat, which means the autopilot is up and listening. + +The first time you run this it may need to download the container image; after that it starts immediately. + +### 4. Back to the first terminal + +Press Enter. The interface script spawns the drones into the scene, waits for MAVLink to come up, and then launches your flight code. + +## Running the interface script + +The interface scripts live in `simulation/interfaces/`. They use imports relative to the `simulation` package, so they must be run **as a module from the repository root**, not as a bare file path: + +```bash +cd /workspace && python -m simulation.interfaces.suas +``` + +Running `python simulation/interfaces/suas.py` instead will fail with an import error. + +The script loads its scene from `simulation/sim_config/`, which is why it has to be started from `/workspace`. + +## Ports + +For drone index `i`, counting from 0: + +| Port | Direction | Set by | +| --- | --- | --- | +| `9003 + 10i` | Unreal to SITL — sensor data | `ardupilot-udp-port` in `sim_config/robot_ardu_quadrotor.jsonc` | +| `9002 + 10i` | SITL to Unreal — servo output | `local-host-udp-port` in the same file | +| `5762 + 10i` | Flight code to the SITL — MAVLink over TCP | the SITL's `--instance` argument | +| `5760 + 10i` | SITL serial0, claimed by MAVProxy | the SITL's `--instance` argument | + +The SITL side of all of these comes from `--instance` alone: the ArduPilot binary adds `10 * instance` to *every* port it uses. The Project AirSim side has to be configured to match. + +> Under Project AirSim these ports live in the `sim_config/*.jsonc` scene and robot configs. The older AirSim `settings.json` mechanism described at the [bottom of this page](#legacy-configuring-airsim-with-settingsjson) is no longer how this is configured. + +## Flying more than one drone + +The `sim` container's default command in `compose.yml` starts a **single** SITL: + +```bash +python /ardupilot/Tools/autotest/sim_vehicle.py -v ArduCopter -f airsim-copter --out=127.0.0.1:14550 -A "--sim-port-in=9002 --sim-port-out=9003" +``` + +To run several drones, override that command in `simulation/compose.override.yml` so the container runs the multi-drone launcher instead, and set `NUM_DRONES` to match: + +```yaml +version: "3" +services: + sim: + command: bash /ardupilot/Tools/autotest/sim_start_drones.sh + environment: + - NUM_DRONES=4 +``` + +`sim_start_drones.sh` starts one SITL per drone in its own tmux window, giving each its own port set using the arithmetic in the table above. Cycle through the windows with `Ctrl-b n` and wait for every one of them to come up before continuing. + +> The drone count has to agree on **both** sides. The interface script and the `sim` container each work out their own port assignments from the number of drones, and nothing checks that the two match. A mismatch shows up as the flight code waiting forever for MAVLink that never arrives. + +## Shutting down + +Stop your flight code, then shut the containers down: + +```bash +./simulation/run_container.sh shutdown +``` + +Then stop **Play** in Unreal. + +To run again, repeat from step 1. The order matters every time — a drone that spawns without its SITL will lock up, and there is no partial recovery. + +## Troubleshooting + +| Symptom | Likely cause | +| --- | --- | +| Connecting to Unreal hangs or is refused | Unreal is not in Play mode, or the wrong host address is set | +| Drones spawn but never move, and the SITL prints `No sensor message received in last 1s, resending servos` | The SITL is not receiving sensor packets from Unreal — check that the two agree on the host address | +| A drone locks up immediately after spawning | Enter was pressed before the SITL was ready. Restart everything; there is no partial recovery | +| `PreArm: Need Position Estimate` for a while | Normal. The EKF is waiting for GPS lock; it clears on its own | +| No MAVLink from the SITL | The number of drones the interface script expects does not match the number of SITLs the `sim` container started | + +If you are running WSL without mirrored networking, Windows and WSL do not share `127.0.0.1`, and you will need to pass the correct host addresses to both sides. Setting `networkingMode=mirrored` in `.wslconfig` avoids this entirely and is strongly recommended. + +## Production + +The final rendition of the simulation is a work in progress. Use the developer workflow above for now. + +## Legacy: configuring AirSim with `settings.json` + +> **This section applies to the older AirSim, not Project AirSim.** Project AirSim is configured through the `.jsonc` scene and robot files in `simulation/sim_config/`, and does not read `settings.json` at all. This is kept for reference and for anyone working with the legacy setup. + +`update_airsim_settings.ps1`, in the root of the simulation repository, copies a settings file to where AirSim expects `settings.json`. Templates live in `simulation/templates/`. See the [Project AirSim settings documentation](https://github.com/iamaisim/ProjectAirSim/blob/main/docs/config.md) or the [AirSim settings documentation](https://microsoft.github.io/AirSim/settings/) for the fields themselves. + +### Single drone ```powershell -.\update_airsim_settings.ps1 .\templates\airsim-settigns-ardupilot.json +.\update_airsim_settings.ps1 .\templates\legacy-airsim-settings-ardupilot.json ``` -If you have a custom settings file: +Or with your own file: -```ps1 +```powershell .\update_airsim_settings.ps1 .\path\to\your\settings.json ``` -#### Multi-Drone Settings +### Multiple drones -If you wish to fly multiple drones, that must be reflected in your settings. Manually doing this is tedious, so our script provides functionality to automate this. For a quick start with 4 drones, you can use the following: +Writing multi-drone settings by hand is tedious, so the script can generate them. For a 2x2 grid of four drones: -```ps1 -.\update_airsim_settings.ps1 .\templates\airsim-settings-multidrone.json -NumDrones 2,2 +```powershell +.\update_airsim_settings.ps1 .\templates\legacy-airsim-settings-multidrone.json -NumDrones 2,2 ``` -The number of drones is given as a grid (`rows,columns`), so the total number of drones is `rows * columns`. +The count is given as a grid (`rows,columns`), so the total is `rows * columns`. -It is important to note that the file provided to the script is used as a *reference* by which a new settigns file is generated. The global settings of the reference is copied as is; the vehicle provided in the reference's `"Vehicles"` section is used as a base for the generated vehicle entries. All settings within the reference vehicle are copied verbatim except for the ports and X, Y, and Z offsets. The port are incremented by `10` from the reference droens for each drone, and the X, Y, and Z are set based on command arguments (which default to 3-meter separation). +The file you pass in is used as a *reference*, not copied verbatim. Global settings are copied as-is, and the single vehicle in the reference's `"Vehicles"` section becomes the template for every generated vehicle. Everything in that vehicle is copied unchanged except the ports and the X, Y, Z offsets: ports increment by `10` per drone, and the offsets come from the separation arguments (3 metres by default). -For instance, given this reference vehicle: +Given this reference vehicle: ```json { @@ -65,7 +206,7 @@ For instance, given this reference vehicle: } ``` -The following drones will be generated: +the script generates: ```json { @@ -106,49 +247,15 @@ The following drones will be generated: } ``` -Any camera settings and such will be carried over for all drones. - -#### More on `.\update_airsim_settings.ps1` - -**Parameters**: - -- `File`: The file to copy to AirSim Settings or use as a reference for generating multidrone settings. Can be explicitly named; otherwise is the first positional argument. -- `OutDir`: The directory to copy the given file to, under the name "settings.json". By default, this is the directory AirSim looks for settigns globally. If using a compiled version of the simulation, you may want to change this to the directory of the executable, since AirSim looks there first. -- `NumDrones`: This is part of a ease-of-use functionality that automatically creates drones compatible with what our simulation Docker container expects. The settings of each drone is copied from the drone settings provided by the given File, but the properties "ControlPort", "UdpPort", "X", and "Y" are adjusted automatically and may be overridden. This parameter expects a grid size input, or ROW,COL. -- `XSep`: Given that NumDrones is provided, determines the x-separation between drones in meters. This parameter is 3 by default. -- `YSep`: Given that NumDrones is provided, determines the y-separation between drones in meters. This parameter is 3 by default. -- `ZOffset`: Given the NumDrones is provided, sets the Z offset for all drones. All drones have the same offset. Negative values correspond to higher altitudes. This parameter is 0 by default. -- `StartControlPort`: Given that NumDrones is provided, determines the control port of the first drone. Each successive drone has a port ten higher than the previous. It is 9002 by default. - - e.g., drone ports will be {9002, 9012, 9022, 9032, ...} if this value is 9002 -- `StartUdpPort`: Given that NumDrones is provided, determines the UDP port of the first drone. Each successive drone has a port ten higher than the previous. It is 9003 by default. - - e.g., drone ports will be {9003, 9013, 9023, 9033, ...} if this value is 9003 - -### Flying - -1. Open your AirSim Unreal Engine project - - you can run AirSim's default project, called *Blocks*, located in your AirSim folder at `\AirSim\Unreal\Environments\Blocks\Blocks.uproject` - - open the uproject directly in Unreal; trying to run blocks from the Visual Studio Solution probably won't work - - if it says that the project was built with a different Unreal version, click "Yes" to rebuild with the version you have -2. Open a terminal and navigate to the root of the `SUAS` repository - - start WSL using the `wsl` command as well -3. Start the Unreal Engine simulation using the editor's `Play` button - - located above the viewport to the far right - - if not immediately visible, press the double-arrow ($>\!\!>$) button, then `Play` - - the Unreal Editor should freeze (this is normal) -4. Start the `sim` and `env` containers using `./run_container.sh` - - **for multi-drone**, you must specify the number of drones using the `NUM_DRONES` environment variable: `NUM_DRONES=4 ./run_container.sh` - - you only need to provide the environment variable when running a command that starts the `sim` container -5. run your code: - - if using the `env` container, attach to the `env` container: `./run_container.sh attach env` - - run your code now - -Whenever you want to rerun code, you must - -- stop the Unreal Simulation -- shutdown the `sim` container - - you can use `./run_container.sh shutdown` to shutdown `env` and `sim` -- repeat steps 3-5 in the above guide +Camera settings and anything else on the reference vehicle carry over to all of them. -## Production +### Script parameters -The final rendition of the simulation is a work in progress! Use the developer stuff for now. +- `File`: The file to copy to AirSim Settings, or use as a reference for generating multi-drone settings. Can be named explicitly; otherwise it is the first positional argument. +- `OutDir`: The directory to copy the file into, under the name `settings.json`. Defaults to the directory AirSim looks in globally. If you are using a compiled build of the simulation you may want to point this at the executable's directory instead, since AirSim looks there first. +- `NumDrones`: Generates drone entries compatible with what the simulation container expects. Each drone's settings are copied from the vehicle in `File`, with `ControlPort`, `UdpPort`, `X`, and `Y` adjusted automatically. Expects a grid size, `ROW,COL`. +- `XSep`: With `NumDrones`, the x-separation between drones in metres. Defaults to 3. +- `YSep`: With `NumDrones`, the y-separation between drones in metres. Defaults to 3. +- `ZOffset`: With `NumDrones`, the Z offset applied to every drone. Negative values are higher altitudes. Defaults to 0. +- `StartControlPort`: With `NumDrones`, the control port of the first drone; each successive drone is ten higher. Defaults to 9002, giving {9002, 9012, 9022, 9032, ...}. +- `StartUdpPort`: With `NumDrones`, the UDP port of the first drone; each successive drone is ten higher. Defaults to 9003, giving {9003, 9013, 9023, 9033, ...}. \ No newline at end of file diff --git a/subteams/simulation/installation/docker/SettingUpDockerImage.md b/subteams/simulation/installation/docker/SettingUpDockerImage.md deleted file mode 100644 index 8d1b96e..0000000 --- a/subteams/simulation/installation/docker/SettingUpDockerImage.md +++ /dev/null @@ -1,4 +0,0 @@ -# How to set up the Ardupilot/Mavlink Docker Image - -## For Ardupilot - diff --git a/subteams/simulation/installation/docker/installing_docker.md b/subteams/simulation/installation/docker/installing_docker.md index d8e5531..c42d7f8 100644 --- a/subteams/simulation/installation/docker/installing_docker.md +++ b/subteams/simulation/installation/docker/installing_docker.md @@ -1,9 +1,9 @@ --- -permalink: /simulation/installation/docker/ +permalink: /simulation/containers/ --- -# Installing And Configuring Docker -Docker is an open-source platform that automates the deployment, scaling, and management of applications using containerization. Containers package an application and its dependencies together, ensuring consistency across different environments. This makes it easier to develop, ship, and run applications reliably on various systems. +# Installing And Configuring Containers +Containers package an application and its dependencies together, ensuring consistency across different environments. This makes it easier to develop, ship, and run applications reliably on various systems. We use [Podman](https://podman.io) rather than [Docker](https://www.docker.com), but the two follow the same [Open Container Initiative](https://opencontainers.org) standards and are largely interchangeable. ## Table of Contents @@ -12,7 +12,7 @@ It is recommended you follow this tutorial in the order listed. - [Download Docker Desktop](#download-docker-desktop) - [Using Windows Subsystem for Linux (WSL)](#using-windows-subsystem-for-linux-wsl) - [Install Host Dependencies](#install-host-dependencies) -- [Building Containers](#building-containers) +- [Getting the Containers](#getting-the-containers) - [Running Containers (Using the Environment)](#running-containers-using-the-environment) - [Configuring Containers](#configuring-the-containers) - [Installing Useful Programming Tools](#installing-useful-programming-tools) @@ -45,24 +45,24 @@ If you will be using our custom Unreal-based simulation (or need to network betw ## Install Host Dependencies -Next, change directories to the repository: +Next, change directories to the repository (make sure to replace with the repo you downloaded): ```bash -cd SUAS-2025 +cd ``` In the repository, we have a script to install the necessary host packages. If you do not need GPU access when coding, run the following: ```bash -./install.sh +./simulation/install.sh ``` If you need GPU access, run the following: ```bash -./install.sh nvidia # if you have an nvidia GPU +./simulation/install.sh nvidia # if you have an nvidia GPU # OR -./install.sh amd # if you have an AMD GPU +./simulation/install.sh amd # if you have an AMD GPU ``` > AMD GPUs are currently not supported (I have an Nvidia GPU, so I don't know the AMD install stuff). If you have an AMD GPU, feel free to add to the installation script and docs! @@ -72,47 +72,57 @@ You will also need the drivers corresponding to your GPU: - Nvidia CUDA Toolkit: - AMD ROCm Install: +This script installs `podman` and `podman-compose`, and then downloads the prebuilt container images, so the next step is usually already done for you by the time it finishes. + **You will need to restart your terminal/shell for the installation to complete.** -## Building Containers +## Getting the Containers + +To simplify (most) of environment setup, we have [containerized](https://en.wikipedia.org/wiki/Containerization_(computing)) our environment. We use a companion command called `podman-compose` (similar to `docker-compose`) that allows us to define how to run our containers in a `compose.yml` file, which lives in the `simulation/` folder. + +**You do not need to build the containers.** We publish prebuilt images to the [GitHub Container Registry](https://github.com/orgs/MissouriMRR/packages), so you download them instead of compiling ArduPilot from source (which takes about 20 minutes). The packages are public, so no login is required. + +`simulation/install.sh` already downloads them for you at the end of the previous step. If you skipped that, or want to re-download them later, run: + +```bash +./simulation/run_container.sh pull +``` -To simplify (most) of environment setup, we have [containerized](https://en.wikipedia.org/wiki/Containerization_(computing)) our environment. Specifically, we use [Podman](https://podman.io), which is pretty much the same as its more popular counterpart, [Docker](https://www.docker.com). (Podman and Docker both follow the [Open Container Initiative](https://opencontainers.org) standards, so they are largely interchangeable.) +**NOTE:** You do not normally have to remember this: `run_container.sh` checks whether an image is present before starting a container, and pulls it automatically if it is missing. -We use a companion command called `podman-compose` (similar to `docker-compose`) that allows us to define how to run our containers in a `compose.yml` file. +### If you need to build them yourself -To build our containers, run the following command from within the `SUAS-2025` folder: +You only need this if you have changed `Env.Containerfile` or `Sim.Containerfile`: ```bash -podman-compose build +./simulation/run_container.sh build ``` This may take a while, so do something else in the meantime. +> If you are the one publishing images for the team, see [Publishing Container Images](/docs/simulation/containers/publishing/). + ## Running Containers (Using the Environment) We have two containers: `env` and `sim`. The `env` container contains everything you need to run your code; the `sim` container will run an [ArduPilot](https://ardupilot.org) drone simulation upon startup. > By default, the `sim` container is meant to be used with the Simulation Subteam's Unreal simulation. If you need to override this, use the `compose.override.yml` file to override the `command` property for the `sim` service to the desired command you can run ([see here](#example-overriding-the-sim-containers-start-command)). If you don't know how compose files work, you can look to `compose.yml` for reference or [read this](https://docs.docker.com/compose/). -For ease of use, we have a `run_container.sh` script. To run both `env` and `sim`, simply run: - -```bash -./run_container.sh -``` +For ease of use, we have a `run_container.sh` script in the `simulation/` folder. It can be run from any directory as it always uses the compose.yml in its directory. -To run the `env` container on its own, run: +Most of the time you want one container attached to your terminal. To start the `env` container and drop into a shell inside it: ```bash -./run_container.sh env +./simulation/run_container.sh env ``` -To attach to the `env` container (connect to it using an interactive shell), run: +To start the `sim` container (which launches the ArduPilot SITL): ```bash -./run_container.sh attach env +./simulation/run_container.sh sim ``` -> Your local SUAS repository code will be mounted in the `env` container, so any changes made to your local copy of the code is automatically reflected in the container, and vice versa. Essentially, the `env` container is a glorified virtual environment. +> Your local repository is mounted inside the `env` container at `/workspace`, so any change to your local copy is immediately reflected in the container, and vice versa. Essentially, the `env` container is a glorified virtual environment. To detach, run the following: @@ -122,16 +132,22 @@ exit > This will also shut down the `env` container; you'll need to start it again. -To shutdown any running containers, do: +To start both containers at once in the background, run the script with no arguments: ```bash -./run_containers.sh shutdown +./simulation/run_container.sh ``` -For more commands, run: +Because these run detached, you will not see their output. Connect to one with: ```bash -./run_containers.sh help +./simulation/run_container.sh attach env +``` + +To shutdown any running containers, do: + +```bash +./simulation/run_container.sh shutdown ``` If you want to take matters into your own hands, you'll need to know how to run/use containers: @@ -146,7 +162,7 @@ If you want to take matters into your own hands, you'll need to know how to run/ ## Configuring the Containers -If you need to configure how a container is run, create a file called `compose.override.yml` in the `SUAS` repository's root directory. This file will allow you to override parameters set in `compose.yml` without modifying up `compose.yml`. If you ran `install.sh` with a GPU selected, `compose.override.yml` should already exist. +If you need to configure how a container is run, create a file called `compose.override.yml` in the `simulation/` folder, next to `compose.yml`. This file will allow you to override parameters set in `compose.yml` without modifying `compose.yml` itself. If you ran `simulation/install.sh` with a GPU selected, `compose.override.yml` should already exist. It is gitignored, so your local settings will not be committed. For more information on compose files, see the following: @@ -158,10 +174,10 @@ For more information on compose files, see the following: By default, the `sim` container is configured to run the following command on startup: ```bash -python /ardupilot/Tools/autotest/sim_vehicle.py -v ArduCopter -f airsim-copter --out=127.0.0.1:14550 +python /ardupilot/Tools/autotest/sim_vehicle.py -v ArduCopter -f airsim-copter --out=127.0.0.1:14550 -A "--sim-port-in=9002 --sim-port-out=9003" ``` -This is the command needed to connect to AirSim (the foundation of the Simulation Subteam's Unreal-based simulation of the SUAS competition). If you need to run a different command, you can use `compose.override.yml`: +This is the command needed to connect to Project AirSim (the foundation of the Simulation Subteam's Unreal-based simulation). If you need to run a different command, you can use `compose.override.yml`: ```yaml version: "3" @@ -191,6 +207,16 @@ services: The above combines the alternate `sim` command with an Nvidia GPU-enabled `env` container. +##### Example: Changing Where Your Code Is Mounted + +The `env` container mounts your competition repository at `/workspace`. By default it mounts the folder that contains `simulation/`, which is correct when the simulation repo is checked out as a submodule at `/simulation`. If your layout is different, set `FLIGHT_CODE_ROOT` in a `.env` file inside the `simulation/` folder: + +``` +FLIGHT_CODE_ROOT=../SUAS-2026 +``` + +`podman-compose` loads `.env` automatically. Like `compose.override.yml`, it is gitignored. + ## Installing Useful Programming Tools #### Visual Studio Code @@ -215,4 +241,4 @@ code in: ```bash code . -``` \ No newline at end of file +``` diff --git a/subteams/simulation/installation/docker/publishing_containers.md b/subteams/simulation/installation/docker/publishing_containers.md new file mode 100644 index 0000000..bb0e36b --- /dev/null +++ b/subteams/simulation/installation/docker/publishing_containers.md @@ -0,0 +1,165 @@ +--- +permalink: /simulation/containers/publishing/ +--- + +# Publishing Container Images + +[Back to Simulation Docs](/docs/simulation/) + +**Most people do not need this page.** If you just want to run the simulation, go to [Installing and Configuring Containers](/docs/simulation/containers/), the images are already published and `run_container.sh` downloads them for you. + +This page is for whoever maintains the images: how to build them, push them to the GitHub Container Registry, and republish them when something changes. + +## Table of Contents + +- [What gets published](#what-gets-published) +- [How a registry actually works](#how-a-registry-actually-works) +- [Publishing a new image for the first time](#publishing-a-new-image-for-the-first-time) +- [Building the images](#building-the-images) +- [Pushing to GHCR](#pushing-to-ghcr) +- [Things that go wrong](#things-that-go-wrong) + +## What gets published + +Two images, both defined in the `simulation/` folder: + +| Image | Built from | Contains | +| --- | --- | --- | +| `ghcr.io/missourimrr/multirotor-env` | `Env.Containerfile` | Python 3.10, the pinned Project AirSim client, `pre-commit` | +| `ghcr.io/missourimrr/multirotor-sim` | `Sim.Containerfile` | ArduPilot compiled from source, plus the SITL launch scripts | + +The `sim` image is the reason this exists at all: building it compiles ArduPilot, which takes roughly 20 minutes. Publishing it once means nobody else pays that cost. + +> Both images build from the `simulation/` folder alone. They do not read anything from the competition repository around them, so a bare clone of the simulation repo is a complete build context. + +## How a registry actually works +``` +ghcr.io /missourimrr/multirotor-sim:latest +└registry┘└─ owner ─┘└─── name ────┘└ tag ┘ +``` + +Tagging an image with a name starting `ghcr.io/...` allows normal git commands (like push, pull) to function normally. Podman builds images through layers, so only the changes you make to each layer are pushed to ghcr.io, vice versa with pulling + +> The owner segment **must be lowercase**. `ghcr.io/MissouriMRR/...` will not work; `ghcr.io/missourimrr/...` will. + +## Publishing a new image for the first time + +Use this when you are adding a *third* image, not when republishing one of the two above. Steps 1 and 2 are one-time setup for your machine and account; everything after that is per image. + +### 1. Get a token and log in + +Create a **classic** personal access token at [github.com/settings/tokens](https://github.com/settings/tokens) with the `write:packages` scope, then log in from WSL: + +```bash +echo "YOUR_PAT_TOKEN" | podman login ghcr.io -u YOUR_GITHUB_USERNAME --password-stdin +``` + +The password field takes the token, not your GitHub account password. You only have to do this once per machine. + +You also need permission to create packages in our organization. If you do not have it and need it, contact your project lead or the CSE for help. + +### 2. Write the Containerfile + +Put it in the `simulation/` folder alongside `Env.Containerfile` and `Sim.Containerfile`, and keep it self-contained. It must not read anything outside that folder, or it cannot be built from a bare clone of the simulation repo. + +End the file with a source label so the published package links back to the repository: + +```dockerfile +LABEL org.opencontainers.image.source=https://github.com/MissouriMRR/Simulation-2026 +``` + +Put it at the *bottom*. A label near the top invalidates every cached layer below it. + +### 3. Build it with its registry name + +The name is the address, so tag it as you build rather than tagging afterwards. Pick a lowercase name that says what the image is: + +```bash +podman build -f My.Containerfile -t ghcr.io/missourimrr/multirotor-mything:latest . +``` + +### 4. Push it + +```bash +podman push ghcr.io/missourimrr/multirotor-mything:latest +``` + +Pushing is what creates the package. There is no separate "create package" step on GitHub. + +### 5. Make the package public + +**New packages are private by default**, and this is the step people forget. The symptom is confusing: teammates get a "not found" error rather than a permissions error. + +1. Go to [github.com/orgs/MissouriMRR/packages](https://github.com/orgs/MissouriMRR/packages). +2. Open the package, then **Package settings**. +3. Under **Danger Zone**, choose **Change visibility** and set it to **Public**. + +Visibility is per package, so a new image needs this even though the existing two are already public. You only do it once, not on every push. + +### 6. Wire it into the scripts + +An image nobody references does nothing. Add it in two places: + +- the `image:` key for its service in `compose.yml` +- the image variables at the top of `run_container.sh` + +There is a comment in each file pointing at the other. If the two disagree, `run_container.sh` pulls a name nothing uses and compose quietly builds instead. + +### 7. Verify it the way a teammate will see it + +Log out and pull with nothing cached: + +```bash +podman logout --all +``` + +```bash +podman rmi ghcr.io/missourimrr/multirotor-mything:latest +``` + +```bash +podman pull ghcr.io/missourimrr/multirotor-mything:latest +``` + +A successful pull while logged out means the package is genuinely public. Deleting the local copy first is safe: your build cache is untouched, so rebuilding is quick if something is wrong. + +## Building the images + +Run these from inside the `simulation/` folder. Tag them with their final registry names as you build, so there is no separate tagging step: + +```bash +podman build -f Env.Containerfile -t ghcr.io/missourimrr/multirotor-env:latest . +``` + +```bash +podman build -f Sim.Containerfile -t ghcr.io/missourimrr/multirotor-sim:latest . +``` + +The `sim` build is the slow one. But the layers are cached, so if you have built it before and only changed something near the bottom of the Containerfile, it will finish in seconds. + +> Both Containerfiles carry an `org.opencontainers.image.source` label, deliberately placed *after* the expensive build steps. A label near the top of the file would invalidate the cached ArduPilot build and turn every rebuild into a 20-minute recompile. If you add labels, add them at the bottom. + +## Pushing to GHCR + +```bash +podman push ghcr.io/missourimrr/multirotor-env:latest +``` + +```bash +podman push ghcr.io/missourimrr/multirotor-sim:latest +``` + +Push the `env` image first when you are testing something — it is smaller, so a mistake surfaces faster. + +> Pushing requires being logged in. If you have not done that on this machine, see [Get a token and log in](#1-get-a-token-and-log-in). + +## Things that go wrong + +| Symptom | Cause | +| --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `unable to retrieve auth token: invalid username/password: unauthorized` | A stale token in podman's saved credentials. Run `podman logout --all` and try again — public packages need no login at all | +| Teammates get "not found" on pull | Go to the repository settings and change the visibility to public | +| `denied` on push | Your token lacks `write:packages`, or you lack package-create permission in the organization | +| Push or pull rejects the name | The owner segment is capitalized. It must be `missourimrr`, all lowercase | +| A trivial change triggers a 20-minute rebuild | Something was inserted near the top of `Sim.Containerfile`, invalidating the cached ArduPilot build. Move it below the `waf` step | +| `podman-compose` builds instead of pulling | Expected. With both `image:` and `build:` set, compose builds when an image is missing rather than pulling. That is why `run_container.sh` pulls explicitly | diff --git a/subteams/simulation/installation/simulation_install_windows.md b/subteams/simulation/installation/simulation_install_windows.md index b3cc3e9..e662512 100644 --- a/subteams/simulation/installation/simulation_install_windows.md +++ b/subteams/simulation/installation/simulation_install_windows.md @@ -1,5 +1,5 @@ --- -permalink: /simulation/installation/windows/ +permalink: /simulation/install/windows/ --- # Simulation Installation and Environment Setup (Windows) @@ -12,21 +12,16 @@ permalink: /simulation/installation/windows/ It is recommended you follow this tutorial in the order listed. -- [Simulation Installation and Environment Setup (Windows)](#simulation-installation-and-environment-setup-windows) - - [Table of Contents](#table-of-contents) - - [Prerequisites](#prerequisites) - - [Installing Unreal Engine](#installing-unreal-engine) - - [Installing Project AirSim](#installing-project-airsim) +- [Installing Unreal Engine](#installing-unreal-engine) +- [Installing Git](#installing-git) +- [Installing Containers](#installing-containers) +- [Installing Project AirSim](#installing-project-airsim) - [Visual Studio](#visual-studio) - [Project AirSim](#project-airsim) - [Environment Setup](#environment-setup) - [Simulation Git Repository](#simulation-git-repository) - [Next Steps](#next-steps) -## Prerequisites - -The installation process below is an extension of flight's installation process, so complete that first (**follow all WSL- and Windows-based options**): [Flight Installation Docs](/docs/flight/installation_guide/). - ## Installing Unreal Engine We will be using Unreal Engine for simulating virtual drones. If you have the Epic Games Launcher installed already (e.g., if you own *Fortnite*), you can download Unreal Engine from the "Unreal Engine" tab. If you don't have the Epic Games Launcher, then you will have to [download](https://store.epicgames.com/en-US/download) it to install Unreal Engine. @@ -35,21 +30,33 @@ The Unreal Engine version you will download is **5.2.1**. Project AirSim, which Once Unreal Engine has finished downloading, you **will need to run it once before doing anything else**. You can continue on to the following steps until told otherwise. +## Installing Git + +Go to the following link and download [Git](https://git-scm.com). The installer should prompt you to set up your credentials needed for downloading GitHub repositories + +## Installing Containers + +We run the flight-code environment and the ArduPilot simulation in containers, managed with Podman inside WSL. Head to our container setup page and follow the instructions [there](/docs/simulation/containers/), then come back here. + ## Installing Project AirSim ### Visual Studio -To install Project AirSim, you must first install Microsoft's [Visual Studio Community 2022](https://visualstudio.microsoft.com). This will also be the IDE you will use for programming C++ code for Unreal. +To install Project AirSim, you must first install Microsoft's [Visual Studio Community 2022](https://drive.google.com/file/d/1lOeKHzdT0Mi3cQxRBX1IQXTz3lq9xBdu/view?usp=drive_link). This will also be the IDE you will use for programming C++ code for Unreal. +- **NOTE:** Microsoft does not keep their older Visual Studio versions on their downloads page, so this is a google drive link to a known working installer. -When installing, you must select the following under the **Individual Components** tab: -- `C++ Development Pack` -- `Windows SDK 10` -- `.NET 8.0 Runtime (Long Term Support)` -- `.NET Framework 4.8 SDK` -- `MSVC v143 - VS 2022 C++ x64/x86 build tools (v14.37-17.7)(Out of support)` +When installing Visual Studio, there are several things you must check for the simulation to work. +- Under **Workloads** (you should start in this page): + - `Desktop development with C++` +- Under **Individual Components** (to the right of **Workloads**): + - `Windows SDK 10` + - `.NET 8.0 Runtime (Long Term Support)` + - `.NET Framework 4.8 SDK` + - `MSVC v143 - VS 2022 C++ x64/x86 build tools (v14.37-17.7)(Out of support)` -You will need to go to `C:\Users\\AppData\Roaming\Unreal Engine\UnrealBuildTool\BuildConfiguration.xml` -- If you cannot find the AppData folder, the easiest way to get there is to press the `Windows` and `R` keys at the same time. This should open a prompt in the bottom left of your screen called `Run`. In the search box type `%appdata%` and press enter. This should put you in your `C:\Users\\AppData\Roaming` folder. +You will need to go to `C:\Users\\AppData\Unreal Engine\UnrealBuildTool\BuildConfiguration.xml` +- If you cannot find the AppData folder, the easiest way it to enable `Hidden Items` in File Explorer. You can do this by going to `view`>`show`>`Hidden Items`. +- If that does not work you can try pressing the `Windows` and `R` keys at the same time. This should open a prompt in the bottom left of your screen called `Run`. In the search box type `%appdata%` and press enter. This should put you in your `C:\Users\\AppData\Roaming` folder. Now replace the contents of BuildConfiguration.xml with the following: ``` @@ -81,11 +88,43 @@ To install Project AirSim, follow these steps: ### Simulation Git Repository -The simulation git repository contains many useful files that streamline running code. Open a terminal (by typing `Terminal` in the Windows search), and `cd C:/path/to/where/you/want/the/repo/at`. +For accessing/editing the actual simulation repository do: +``` +git clone https://github.com/MissouriMRR/Simulation.git +``` -Now run `git clone https://github.com/MissouriMRR/Simulation-2023.git`. This will create a new directory called `\Simulation-2023\` +The simulation code lives as a submodule inside the competition repository, so clone the competition repo *with submodules* rather than cloning the simulation repo on its own. +Open a terminal (by typing `Terminal` in the Windows search), `cd` to where you want the repository, and run: + +``` +git clone --recurse-submodules https://github.com/MissouriMRR/.git +``` + +This creates a `\` directory, with the simulation code at `\simulation\`. If you have already cloned it without `--recurse-submodules`, run this from inside the repository to fill the submodule in: + +``` +git submodule update --init --recursive +``` +> The containers run inside WSL, so clone somewhere WSL can reach. A path on your Windows drive is fine — WSL sees it under `/mnt/c/...`. + +### Simulation Unreal Repository + +We have our own self-hosted GitHub repository on the bay computer (Ricky), to get access to you will need to do a few things: +1. Run ```ssh-keygen -t ed25519 -C "your_email@example.com"``` in a terminal. + 1. You can press 'Enter' 3x to skip through setting whether you want the file to require a passphrase. Or you can set it, just don't forget it + 2. Run ```cat ~/.ssh/id_ed25518.pub``` in a terminal + 3. Copy the output to your clipboard +2. Message this to the Simulation Project Lead or Special Projects Lead with context, and they should be able to get you set up +3. Once they give you the go-ahead, you will need to be on the school's network. + 1. If you are not, you can still access it, you just need to download the school's VPN + 2. At `https://openvpn.mst.edu/` download the correct OpenVPN client for your computer + 3. Also download the latest .ovpn file under `Configuration Files` + 4. Once both have downloaded and you have installed the OpenVPN software, there will be a button to upload a file + 5. Press said button and select the .ovpn file you downloaded earlier + 6. You are now on the school's WiFi! +4. Now you should be able to download the repository with `git clone git@131.151.19.149:SimulationRepository.git` ## Next Steps diff --git a/subteams/simulation/simulation.md b/subteams/simulation/simulation.md index 11d6c6f..a858596 100644 --- a/subteams/simulation/simulation.md +++ b/subteams/simulation/simulation.md @@ -21,7 +21,9 @@ After you have installed the simulator, proceed to [Flying the Drone with Code]( ## Docs Directory -- [Simulation Installation](/docs/simulation/installation/) +- [Simulation Installation](/docs/simulation/installation/simulation_install) +- [Installing and Configuring Containers](/docs/simulation/containers/) +- [Publishing Container Images](/docs/simulation/containers/publishing/) - [Flying the Drone with Code](/docs/simulation/flying/) - [Using RealityScan to Create 3-D Models](/docs/simulation/realityscan) - [Run Sim On Pi](/docs/simulation/raspberry_pi_sim/)