Pose2Sim

Pose2Sim.mp4

Pose2Sim provides a workflow for 3D markerless kinematics (human or animal), as an alternative to traditional marker-based MoCap methods.

Pose2Sim is free and open-source, requiring low-cost hardware but with research-grade accuracy and production-grade robustness. It gives maximum control over clearly explained parameters. Any combination of phones, webcams, or GoPros can be used with fully clothed subjects, so it is particularly adapted to the sports field, the doctor's office, or for outdoor 3D animation capture.

Tip

For real-time analysis with a single camera, please consider Sports2D (note that the motion must lie in the sagittal or frontal plane).

Note

Features:

📹 Any Cameras (phones, webcams, GoPros...)
🎯 Research-Grade Accuracy (validated by peer-reviewed studies)
👥 Multi-Person Support (track multiple people simultaneously)
🤸 Full 3D kinematics (complete OpenSim skeletal analysis with joint angles)

Use cases:

🏀 Sports Analysis (field-based 3D motion capture)
🏥 Clinical Assessment (gait analysis in doctor's office)
🎭 Animation (outdoor 3D capture with fully clothed subjects)
🔬 Research (biomechanics studies with multiple participants)

Note

Fun fact: Pose2Sim stands for "OpenPose to OpenSim", as it originally used OpenPose inputs (2D keypoints coordinates) and led to an OpenSim result (full-body 3D joint angles). Pose estimation is now performed with more recent models from RTMPose, and custom models (from DeepLabCut, for example) can also be used.

Important

If you like it, ⭐ please leave a star ⭐ on the GitHub repository!
This project is completely free: this is your chance to support the project and make it more visible to the community.

Important

If you want to contribute to Sports2D or Pose2Sim, please see How to contribute or join the Discord community!

Installation and Demonstration

Installation

Install Pose2Sim

Note

If you'd rather use conda, you can still use the old installation procedure. Still works fine but not recommended, since uv is faster, lighter, better at handling dependencies, and generally more modern.

1. Set up a uv environment:

Open a terminal (conda, powershell, bash, or zsh).

On Windows:

  # Install uv
  powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
  # Create uv environment
  uv venv "$env:USERPROFILE\.venv\pose2sim" --python 3.13 # or 3.11, or 3.12 
  # Activate the uv environment
  & "$env:USERPROFILE\.venv\pose2sim\Scripts\Activate.ps1"

On Linux or MacOS:

  # Install uv
  curl -LsSf https://astral.sh/uv/install.sh | sh
  # Create uv environment
  uv venv ~/.venv/pose2sim --python 3.13 # or 3.11, or 3.12 
  # Activate the uv environment
  source ~/.venv/pose2sim/bin/activate

Tip

Remembering the command for activating the uv environment can be a pain. Just type Ctrl+R in your terminal and start typing activate to find it.

2. Install Pose2Sim:

Open a terminal (conda, powershell, bash, or zsh).
Activate your environment (see here).

OPTION 1: Latest stable version:
```
  uv pip install pose2sim --upgrade
```

OPTION 2: For developers who want to test and edit the bleeding edge version:

  git clone --depth 1 https://github.com/perfanalytics/pose2sim.git
  cd pose2sim
  uv pip install -e .

3. Optional:

For faster inference, you can run on the GPU.
Be aware that GPU support takes almost 5 GB more on disk.

Open a terminal, activate your environment (see here), and run nvidia-smi. If this results in an error, your GPU is probably not compatible with CUDA. If not, note the "CUDA version": it is the latest version your driver is compatible with (more information on this post).

Then go to the ONNXruntime requirement page, note the latest compatible CUDA and cuDNN requirements. Next, go to the pyTorch website and install the latest version that satisfies these requirements (beware that torch 2.4 ships with cuDNN 9, while torch 2.3 installs cuDNN 8). For example:

  uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

Finally, install ONNX Runtime with GPU support:

  uv pip uninstall onnxruntime
  uv pip install onnxruntime-gpu

Check that everything went well within Python with these commands:

  import torch; import onnxruntime as ort
  print(torch.cuda.is_available(), ort.get_available_providers())
  # Should print "True ['CUDAExecutionProvider', ...]"

Tip

Note on storage use:
A full installation takes up to 14 GB of storage space. HoweveGPU and GUI supports are not mandatory and take about 5 GB. Tcache can be cleared to save space if you don't care for futuinstallations to be instantaneous. A minimal installation with carefully chosen pose models and without GPU support would take less than 2 GB.

Demonstration Part-1: End to end video to 3D joint angle computation

Test the full pipeline!

Open a terminal (conda, powershell, bash, or zsh) and activate your environment (see here).
Find the Demo folder under <pose2sim_path>\Pose2Sim\Demo_SinglePerson:
```
uv pip show pose2sim # to find <pose2sim_path>
```
Copy-paste the Demo folder wherever you like, and rename it as you wish (manually or using the cp command).
Go to the Demo_SinglePerson folder and start python:
```
cd <Demo_SinglePerson_path>
ipython
```

Try the following code:

  from Pose2Sim import Pose2Sim
  Pose2Sim.calibration()
  Pose2Sim.poseEstimation()
  Pose2Sim.synchronization()
  Pose2Sim.personAssociation()
  Pose2Sim.triangulation()
  Pose2Sim.filtering()
  Pose2Sim.markerAugmentation()
  Pose2Sim.kinematics()

Note

3D marker trajectories are stored as .trc files in each trial folder in the pose-3d directory.
3D joint angles are stored as .mot files in the kinematics directory. Scaled .osim models are also stored in the same directory.
Processing details and statistics are stored in logs.txt.

Demonstration Part-2: Visualize your results with OpenSim or Blender

Visualize your results and look in detail for potential areas of improvement.

Basic visualization with the OpenSim GUI

Install OpenSim GUI:
Download the executable there.
Visualize results:
- Open the OpenSim GUI, go to File > Open Model, and select the scaled model in the kinematics folder.
- Go to File > Load Motion, and load the joint angle .mot file in the kinematics folder.
- If you want to see the 3D marker locations, go to File > Preview Experimental Data, and load the .trc file in the pose-3d folder.

Further investigation with the Pose2Sim Blender add-on

Install the add-on:
Follow instructions on the Pose2Sim_Blender add-on page.
Visualize results:
Just play with the buttons!
Visualize camera positions, videos, triangulated keypoints, OpenSim skeleton, video overlay your results on videos, ... or let your creativity flow and create your own animations!

Pose2Sim_Blender_short.mp4

Demonstration Part-3: Try multi-person and batch analyses

Try multi-person and batch analyses

Open a terminal (conda, powershell, bash, or zsh) and activate your environment (see here).

Multi person analysis

Discover another person, hidden all along!

Similarly to Part-1, find the Multi-Person Demo folder under <pose2sim_path>\Pose2Sim\Demo_MultiPerson, and move it and rename it if you like. Make sure you set multi_person = true in your Config.toml file. Go to the <Demo_MultiPerson> folder and start python:

  cd <Demo_MultiPerson_path>
  ipython
  from Pose2Sim import Pose2Sim
  Pose2Sim.runAll()

Note

Pose2Sim uses sophisticated algorithms to Match people across views, Track IDs over time, Handle occlusions.

Batch processing

Scale from research studies to production pipelines!
Run numerous analyses with different parameters and minimal friction.

Find the Batch Demo folder under <pose2sim_path>\Pose2Sim\Demo_Batch, and move it and rename it if you like. Go to the <Demo_Batch> folder and start python:

cd <Demo_Batch_path>
ipython
from Pose2Sim import Pose2Sim
Pose2Sim.runAll()

Note

This lets you run entire sessions with consistent calibration and global settings across trials.
Trial-specific customization is possible when needed

Tip

The batch processing structure requires a root Config.toml file, as well as one in each of the trial directories. Global parameters are given in the root one. Any trial can override specific parameters by uncommenting keys and their values in the trial-specific Config.toml files.

For example, try uncommenting [project] and set frame_range = [10,99] in the root Config.toml, and uncomment [pose] and set mode = 'lightweight' in the Trial_2 one.

Run Pose2Sim from the <Demo_Batch_path> folder if you want to batch process the whole session, or from a subfolder if you only want to process a specific trial.

SingleTrial

BatchSession

SingleTrial                    
├── calibration
├── videos
└── Config.toml

BatchSession                     
├── calibration 
├── Trial_1   
│   ├── videos 
│   └── Config.toml # Trial specific parameters
├── Trial_2 
│   ├── videos 
│   └── Config.toml # Different trial parameters
└── Config.toml # Global parameters

Demonstration Part-4: Go further

Playing with parameters and try different features!

Default parameters are provided in Config.toml but can be edited.
All of them are clearly documented: feel free to play with them!

Try the calibration tool:
Set calibration_type to calculate in Config.toml and follow the process (more info there).
```
Pose2Sim.calibration()
```

Pass an updated config dictionary instead of editing Config.toml, and test whole_body pose estimation:

import toml
config_dict = toml.load("<Demo_SinglePerson_path>/Config.toml")
config_dict.get("project").update({"project_dir":"<Demo_SinglePerson_path>"})
config_dict.get("pose").update({"pose_model": "whole_body","overwrite_pose": True})
Pose2Sim.poseEstimation(config_dict)

# Or even simpler, just pass the updated parameters
config_dict = {"project": {"project_dir": "<Demo_SinglePerson_path>"}, 
               "pose": {"pose_model": "whole_body", "overwrite_pose": True}}
Pose2Sim.poseEstimation(config_dict)

Run all stages at once:

from Pose2Sim import Pose2Sim
Pose2Sim.runAll()
# or: 
# Pose2Sim.runAll(do_calibration=True, do_poseEstimation=True, do_synchronization=True, do_personAssociation=True, do_triangulation=True, do_filtering=True, do_markerAugmentation=True, do_kinematics=True)

Too slow for you?

Pose2Sim can probably run 10 - 20 times faster with a few tweaks

Project set up:

Set multi_person = True in your Config.toml file, and remove all the detections other than the ones of interest. It works as well as the single person mode, and is much faster. In the future, I plan to add a person selection feature (like on Sports2D) and to remove the old single person mode.
Limit frame_range to only process a subset of frames.

Pose2Sim.calibration():
Run it only when your cameras are moved or changed. If they are not, just copy a previous calibration.toml file into your new calibration folder.

Pose2Sim.poseEstimation():

Use your GPU: This makes pose estimation significantly faster, without any impact on accuracy. See Installation section for more information.
1 min 23 s -> 38 s on my average laptop
Set display_detection = false. Do not display results in real time.
38 s -> 30 s
Set parallel_workers_pose = 'auto' or an integer (number of threads). 'auto': one thread per video (ie 4 on the Demo data). This requires display_detection = false. No impact on accuracy either.
30 s -> 19 s
Set det_frequency = 100 in Config.toml. Run the bounding box detector and the pose estimator on the first frame; for all subsequent frames, only run pose estimation. No impact on accuracy but may miss detection or swap some person IDs if several persons are in the scene.
19 s -> 9 s
Use mode = 'lightweight': Will use a lighter version of RTMPose, which is faster but less accurate.
9 s -> 6.5 s
Set save_video = 'none'. Do not save images and videos
6.5 s -> 5 s
Set tracking_mode = 'sports2d' or tracking_mode = 'none'. If several persons are in the scene, use the sports2d tracker or no tracker at all, but not 'deepsort' (sports2d tracking is almost instantaneous though).

Pose2Sim.synchronization():
Do not run if your cameras are natively synchronized.

Pose2Sim.personAssociation():
Do not run if there is only one person in the scene.

Pose2Sim.triangulation():
Not much to do here.

Pose2Sim.filtering():
You can skip this step, but it is quite fast already.

Pose2Sim.markerAugmentation():
Very fast, too. Note that marker augmentation won't necessarily improve results so you can consider skipping it.

Pose2Sim.kinematics():

Set use_simple_model = true. Use a simpler OpenSim model, without muscles and constraints. Note that the spine will be stiff and shoulders will be a simple ball joint, but this is accurate enough for most gait-related tasks
9 s -> 0.7 s
Set parallel_workers_kinematics = 'auto' or an integer (number of processes). Only works in multi-person mode.

Use on your own data

Setting up your project

Get yourself comfy!

Tip

If any of the following steps is not relevant for your use case (synchronization, person association, marker augmentation...), you can just skip it.

Open a terminal, enter uv pip show pose2sim, report package location.
Copy this path and do cd <path>\pose2sim.
Copy the appropriate Demo folder as your project template (Demo_SinglePerson, Demo_MultiPerson, or Demo_Batch).
Paste it to your preferred location and rename as desired.
Edit the parameters in the [project] section of Config.toml, e.g. project_dir, participant_height, participant_mass, etc.
Add your videos to the videos folder. We recommend renaming them for clarity.
The rest of the tutorial will explain to you how to populate the Calibration and videos folders, run each Pose2Sim step, and edit the Config.toml file.

Initial project structure:

SingleTrial

BatchSession

SingleTrial                    
├── calibration
├── videos
└── Config.toml

BatchSession                     
├── calibration 
├── Trial_1   
│   ├── videos 
│   └── Config.toml # Trial specific parameters
├── Trial_2 
│   ├── videos 
│   └── Config.toml # Different trial parameters
└── Config.toml # Global parameters

Note

Uncluttered background and good lighting conditions will help you get better results. Note that tripods may sometimes be mistaken for people.

Tip

Consider using the Lab Camera Optimizer tool to help you position your cameras in difficult settings.

2D pose estimation

Estimate 2D pose from images with RTMPose or another pose estimation solution.

With RTMPose (default):

RTMPose is a state-of-the-art pose estimation solution that is accurate, fast, flexible, and natively integrated in Pose2Sim.

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.poseEstimation()

This will run pose estimation on your videos and save the results in the pose folder. They are provided in the OpenPose format, with one .json file per frame per camera, containing keypoint coordinates and confidence scores.

Tip

See the Too slow for you? section to make it faster by deactivating real-time display, running parallel pose estimation, using your GPU, increasing the detection frequency, and using a lighter model.

Tip

Other available models:

Whole body: Use 'Whole_body' pose_model in Config.toml to track 133 keypoints on the body, face, and hands. Note that it is slower and slightly less accurate than the default 'Body_with_feet' model on body keypoints.
Wrist motion: Use 'Whole_body_wrist' pose_model (effectively runs the 'whole_body' model) but ignores all face and finger keypoints, except 2 per hand.
Lower body: Use 'Lower_body' pose_model. Effectively runs the 'body_with_feet' model but ignores all upper body keypoints.
Others: You can also use the Hand, Face, Animal, or any other RTMlib model (see next tip).

Tip

To use custom detection and pose models:

Other than 'lightweight', 'balanced', or 'performance' modes, you can use any other pose estimation models through RTMLib (hand, face, animal, or any custom trained models).
The (optional) detection model, pose model, and input sizes can be written in a dictionary (within triple quotes) as shown below. Models can be local paths or URLs, with .onnx or .zip extensions. Make sure the input_sizes are within square brackets.

# Equivalent to mode='balanced', with body_with_feet pose model
mode = """{'det_class':'YOLOX',
       'det_model':'https://download.openmmlab.com/mmpose/v1/prects/rtmposev1/onnx_sdk/yolox_m_8xb8-300e_humanart-c2c7a14a.zip',
       'det_input_size':[640, 640],
       'pose_class':'RTMPose',
       'pose_model':'https://download.openmmlab.com/mmpose/v1/prects/rtmposev1/onnx_sdk/rtose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.zi,
       'pose_input_size':[192,256]}"""

# With one-stage RTMO model 
# Requires pose_model = 'Body'. Marker augmentation won't work, Kimatic analysis will
mode = """{'pose_class':'RTMO', 
       'pose_model':'https://download.openmmlab.com/mmpose/v1/prects/rtmo/onnx_sdk/rt-m_16xb16-600e_body7-640x640-39e78cc4_20231211.zip', 
       'pose_input_size':[640, 640]}"""

# With animal pose estimation:
# Marker augmentation won't work, and you will need to create your owOpenSim skeleton for kinematic analysis.
mode = """{'pose_class':'RTMPose',
       'pose_model':'https://download.openmmlab.com/mmpose/v1/prects/rtmposev1/onnx_sdk/> rtose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.zip',
       'pose_input_size':[256,256]}"""

# Same approach for hand or face pose estimation, check the RTMLib domentation for more information.

Warning

Note that it does not currently work well for acrobatic movements where the person is upside down. We are working on a solution but in the meantime, you can try MediaPipe BlazePose (see here).

With MMPose (coming soon):

Coming soon

With DeepLabCut:

If you need to detect specific points on a human being, an animal, or an object, you can also train your own model with DeepLabCut. In this case, Pose2Sim is used as an alternative to AniPose.

Train your DeepLabCut model and run it on your images or videos (more instruction on their repository)
Translate the h5 2D coordinates to json files (with DLC_to_OpenPose.py script, see Utilities). Note that the names of your camera folders must follow the same order as in the calibration file, and end with '_json':

DLC_to_OpenPose -i input_h5_file

Edit pose.CUSTOM in Config.toml, and edit the node IDs so that they correspond to the column numbers of the 2D pose file, starting from zero. Make sure you also changed the pose_model and the tracked_keypoint.
You can visualize your skeleton's hierarchy by changing pose_model to CUSTOM and writing these lines:

config_path = r'path_to_Config.toml'
import toml, anytree
config = toml.load(config_path)
pose_model = config.get('pose').get('pose_model')
model = anytree.importer.DictImporter().import_(config.get('pose').get(pose_model))
for pre, _, node in anytree.RenderTree(model): 
    print(f'{pre}{node.name} id={node.id}')

Create an OpenSim model if you need inverse kinematics.

With OpenPose (legacy):

Warning

RTMlib is faster, more accurate, and easier to install than OpenPose. This is a legacy option.

Warning

OpenPose model files are apparently not available on their website anymore. Send me an email at contact@david-pagnon.com if you want me to forward them to you!

The accuracy and robustness of Pose2Sim have been thoroughly assessed only with OpenPose, BODY_25B model. Consequently, we recommend using this 2D pose estimation solution. See OpenPose repository for installation and running. Windows portable demo is enough.

Open a command prompt in your OpenPose directory.
Launch OpenPose for each videos folder:

bin\OpenPoseDemo.exe --model_pose BODY_25B --video <PATH_TO_TRIAL_DIR>\videos\cam01.mp4 --write_json <PATH_TO_TRIAL_DIR>\pose\pose_cam01_json

The BODY_25B model has more accurate results than the standard BODY_25 one and has been extensively tested for Pose2Sim.
You can also use the BODY_135 model, which allows for the evaluation of pronation/supination, wrist flexion, and wrist deviation.
All other OpenPose models (BODY_25, COCO, MPII) are also supported.
Make sure you modify the Config.toml file accordingly.
Use one of the json_display_with_img.py or json_display_with_img.py scripts (see Utilities) if you want to display 2D pose detections.

With MediaPipe BlazePose (legacy):

Warning

RTMlib is faster, more accurate, and easier to install than BlazePose. This is also a legacy option.

Mediapipe BlazePose is very fast, fully runs under Python, handles upside-down postures and wrist movements (but no subtalar ankle angles).
However, it is less robust and accurate than OpenPose, and can only detect a single person.

Use the script Blazepose_runsave.py (see Utilities) to run BlazePose under Python, and store the detected coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format:
```
Blazepose_runsave -i input_file -dJs
```
Type in Blazepose_runsave -h for explanation on parameters.
Make sure you changed the pose_model and the tracked_keypoint in the Config.toml file.

With AlphaPose (legacy):

Warning

RTMlib is faster, more accurate, and easier to install than AlphaPose. This is also a legacy option.

AlphaPose is one of the main competitors of OpenPose, and its accuracy is comparable. As a top-down approach (unlike OpenPose which is bottom-up), it is faster on single-person detection, but slower on multi-person detection.
All AlphaPose models are supported (HALPE_26, HALPE_68, HALPE_136, COCO_133, COCO, MPII). For COCO and MPII, AlphaPose must be run with the flag "--format cmu".

Install and run AlphaPose on your videos (more instruction on their repository)
Translate the AlphaPose single json file to OpenPose frame-by-frame files (with AlphaPose_to_OpenPose.py script, see Utilities):
```
AlphaPose_to_OpenPose -i input_alphapose_json_file
```
Make sure you changed the pose_model and the tracked_keypoint in the Config.toml file.

Camera calibration

Calculate camera intrinsic properties (lens characteristics) and extrinsic parameters (positions and orientations).
Convert a preexisting calibration file, or calculate intrinsic and extrinsic parameters from scratch.

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.calibration()

Output file:

Convert from Caliscope, AniPose, FreeMocap, Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV

If you already have a calibration file, set calibration_type type to convert in your Config.toml file.

Note

If the original calibration file does not provide any residual errors, they will be logged as NaN. This is not an error and can be ignored.

From Caliscope (recommended), Dynamic Extrinsic Camera Calibrator, AniPose or FreeMocap:
- Copy your .toml calibration file to the Pose2Sim Calibration folder.
- Calibration can be skipped since these formats are natively supported by Pose2Sim.
- Note: It seems like the FreeMoCap calibration is in millimeters rather than in meters. Just open your calibration.toml file and multiply all the translation values by 1000.
From Qualisys:
- Export calibration to .qca.txt within QTM (see there).
- Copy it in the Calibration Pose2Sim folder.
- set convert_from to 'qualisys' in your Config.toml file. Change binning_factor to 2 if you film in 540p.
- If you set your cameras vertically and the videos are rendered sideways, you need to rotate them and the calibration file before running pose estimation. Use this script.
From Optitrack: Exporting calibration will be available in Motive 3.2. In the meantime:
- Calculate intrinsics with a board (see next section).
- Use their C++ API to retrieve extrinsic properties. Translation can be copied as is in your Calib.toml file, but TT_CameraOrientationMatrix first needs to be converted to a Rodrigues vector with OpenCV. See instructions here.
- Use the Calib.toml file as is and do not run Pose2Sim.calibration()
From Vicon:
- Copy your .xcp Vicon calibration file to the Pose2Sim Calibration folder.
- set convert_from to 'vicon' in your Config.toml file. No other setting is needed.
From OpenCap:
- Copy your .pickle OpenCap calibration files to the Pose2Sim Calibration folder.
- set convert_from to 'opencap' in your Config.toml file. No other setting is needed.
From EasyMocap:
- Copy your intri.yml and extri.yml files to the Pose2Sim Calibration folder.
- set convert_from to 'easymocap' in your Config.toml file. No other setting is needed.
From bioCV:
- Copy your bioCV calibration files (no extension) to the Pose2Sim Calibration folder.
- set convert_from to 'biocv' in your Config.toml file. No other setting is needed.

Calculate from scratch

Calculate calibration parameters with a checkerboard, with measurements on the scene, or automatically with detected keypoints.
Take heart, it is not that complicated once you get the hang of it!

Tip

Try the calibration tool on the Demo by changing calibration_type to calculate in Config.toml.
For the sake of practicality, there are voluntarily few board images for intrinsic calibration in our Demo, and few points to click for extrinsic calibration. In spite of this, your reprojection error should be under 1-2 cm, which does not hinder the quality of kinematic results in practice.

1. Calculate intrinsic parameters with a checkerboard

Note

Intrinsic parameters: camera properties (focal length, optical center, distortion).
They are stored in a Calib.toml file: focal length in matrix[0,0] and matrix[1,1], optical center in matrix[0,2] and matrix[1,2], distortion in the distortions array.

For each camera, film a checkerboard or a charucoboard. Either the board or the camera can be moved.
Create a folder for each camera in your Calibration\intrinsics folder and copy your images or videos in them.
Adjust intrinsics_corners_nb and intrinsic_square_size in Config.toml.

Tip

The intrinsic parameters usually need to be calculated only once in their lifetime. In theory, cameras with the same model and settings will have identical intrinsic parameters, so they can be copied from one Calib.toml file to another. In practice, small variations can occur.
If you already calculated intrinsic parameters earlier, you can skip this step by setting overwrite_intrinsics to false.

Tip

Checkerboard requirements:

Flat: Board must be completely flat
Asymmetric: Rows ≠ Columns (or rows odd if columns even)
Border: Wide white border around pattern
Focus: Sharp, in-focus images
Coverage: Film from multiple angles covering most of frame
No glare: Avoid reflections Generate checkerboard at calib.io

Important

Common errors:

Specifying the external, instead of the internal number of corners (one less than the count from calib.io). This may be one less than you would intuitively think.
Taking photos from the scene instead of extracting frames from a video. The photo image format is often different from the video one, which skews intrinsic calibration.

Important

Intrinsic calibration error should be below 0.5 px.

2.Calculate extrinsic parameters

Note

Extrinsic parameters: camera positions and orientations in space, need to be recalculated whenever a camera is moved. Can be calculated from a board, or from points in the scene with known coordinates.
They are stored in a Calib.toml file, in the translation and rotation arrays.

Tip

If there is no measurable item in the scene, you can temporarily bring something in (a table, for example), perform calibration, and then remove it before you start capturing motion.

3 available methods:
- With a checkerboard:
  Make sure that it is seen by all cameras.
  Can be set horizontally (default) or vertically (set board_position = 'vertical' in Config.toml).
  It should preferably be rather large, as results will not be very accurate out of the covered zone.
  Adjust extrinsics_corners_nb and extrinsic_square_size.
- With scene measurements (more flexible and potentially more accurate if points are spread out):
  Manually measure the 3D coordinates of 10 or more points in the scene (tiles, wall lines, boxes, treadmill dimensions...). These points should be as spread out as possible. Replace object_coords_3d by these coordinates in Config.toml.
  Then you will be prompted to click on the corresponding image points for each view.
- With keypoints:
  For a more automatic calibration, pose keypoints could also be used for calibration.
  COMING SOON!
Once your cameras are in place, make a quicke recording of the checkerboard laid on the floor, or the raw scene (only one frame is needed, but do not just take a photo unless you are sure it does not change the image format).
You can remove the checkerboard or the calibration object for the actual capture of your participants.
Copy your files in the extrinsics folder.
Adjust parameters in the Config.toml file.

Important

Extrinsic calibration error should be below 1 cm, but depending on your application, results will still be potentially acceptable up to 2.5 cm.

Synchronization

2D points can be triangulated only if they represent the same body position across all cameras: therefore, views need to be synchronized. This module helps you do it.
For each camera, the algorithm computes mean vertical speed for the chosen keypoints, and synchronizes by finding the time offset for which the correlation is highest.

Tip

Skip this step if your cameras are natively synchronized.

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.synchronization()

You can choose the keypoints to synchronize on, the reference person, and the time window for analysis. You can either tune these parameters in the GUI (set synchronization_gui = true) or set them in your Config.toml file.

Tip

Works best when:

the participant does not move towards or away from the cameras
they perform a clear vertical movement
the capture lasts at least 5 seconds, so that there is enough data to synchronize on
the capture lasts a few minutes maximum, so that cameras are less likely to drift with time

Note

Alternatively, synchronize cameras using a flashlight, a clap, or a clear visual or audio event. The speed of sound being lesser than the speed of light, visual events should be preferred.

Tip

GoPro cameras can also be synchronized with a timecode or by GPS (outdoors).

Associate persons across cameras

If multi_person is set to false, the algorithm chooses the person for whom the reprojection error is smallest.
If multi_person is set to true, it associates persons across views by measuring how closely the 3D rays from cameras to keypoints intersect. In the triangulation stage, people are then associated across frames according to their displacement speed.

Tip

Skip this step if only one person is visible during the capture.

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.personAssociation()

Tip

If persons are frequently lost, raise reconstruction_error_threshold, lower min_affinity, or lower min_cameras_for_triangulation ([triangulation] section) in your Config.toml file.
If persons are wrongly associated, change the same parameters in the opposite direction.

Note

Pose2Sim is robust to the losses of persons in some camera views (e.g., due to occlusions or to the person entering/leaving the scene).

Triangulating keypoints

Robustly triangulate your 2D coordinates:

The triangulation is weighted by the likelihood of each detected 2D keypoint, provided that their likelihood is above likelihood_threshold_triangulation.

If the reprojection error is above reproj_error_threshold_triangulation, cameras are progressively excluded until the threshold is met. If more cameras are removed than min_cameras_for_triangulation, triangulation is skipped for this point and this frame.

A person is given a new ID if they are not seen within max_distance_m m of their previous position and lost for more than max_unseen_frames frames.

In the end, gaps smaller than interp_if_gap_smaller_than frames are interpolated. See Config.toml file for more triangulation parameters.

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.triangulation()

The logs provide detailed statistics:

For each keypoint, their mean reprojection error (mm and px), the average number of cameras excluded, and the interpolated frames.
For the full keypoint set, the grand mean reprojection error (mm and px), the grand mean number of cameras excluded, and the percentage of exclusion for each camera.

Tip

Visualize your output trc files in the Blender add-on or in OpenSim: File -> Preview experimental data.

Tip

If results are not satisfactory, primarily increase min_cameras_for_triangulation in your Config.toml file.
Then try increasing likelihood_threshold_triangulation or decreasing reproj_error_threshold_triangulation.

Filtering 3D coordinates

Filter your 3D coordinates. Available filters:

Butterworth: Most intuitive and standard filter in biomechanics

Acceleration minimizing: Whittaker-Henderson filter, produces smooth velocities and accelerations, which is important for force estimations but blunts the peaks.

Kalman: Used in countless applications, especially real-time. This simplified Kalman filter assumes constant acceleration with Gaussian process noise.

OneEuro: Simpler and even faster alternative to Kalman filter for real-time, but tends to blunt RoM. Analog to a 1st order Butterworth but with adaptive cut-off frequency. Zero-phase in our case

GCV spline: Automatically determines optimal cut-off frequency for each point, which is good when some move faster than others (eg fingers vs hips). The cut-off frequency determination can be buggy if the signal is a triangular wave + drift, for vertical hip coordinates in uphill gait for example.

LOESS: Local low-degree polynomial smoothing.

Gaussian: Weighted moving average, with more weight on central points.

Median: Removes outliers (but not as well as the Hampel filter).

Butterworth on speed: Signal is differenciated, filtered, then integrated back. Preserves sharp movements, but can introduce an offset in the original signal. They can all be tuned with the parameters in your Config.toml file.

A Hampel filter can optionally be applied by setting reject_outliers to true. Rejects outliers that are outside of a 95% confidence interal from the median in a sliding window of size 7.

Tip

Instead of, or in addition to filtering triangulated trc coordinates, you can also filter angle .mot files after inverse kinematics by setting filter_ik = true in your Config.toml file.

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.filtering()

Check your filtration with the displayed figures, and visualize your .trc file in OpenSim. If your filtering is not satisfying, try and change the parameters in the Config.toml file.

Output:

Marker Augmentation

Use the Stanford LSTM model to estimate the position of 47 virtual markers.

Tip

Marker augmentation tends to give a more stable, but less precise output. In practice, it is mostly beneficial when using fewer than 4 cameras. Skip if inverse kinematic results are not convincing.

Important

Make sure that participant_height is correct in your Config.toml file
participant_mass is mostly optional for IK
Only works with models estimating at least the following keypoints (e.g., not COCO):[RHip, LHip, RKnee, LKnee, RAnkle, LAnkle, RHeel, LHeel, RSmallToe, LSmallToe, RBigToe, LBigToe]
Will not work properly if missing values are not interpolated (i.e., if there are NaN values in the .trc file)

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.markerAugmentation()

OpenSim kinematics

Scale an OpenSim skeletal model to your participant and compute biomechanically accurate 3D joint angles using inverse kinematics.

Tip

If you are not interested in muscles or having a flexible spine, set use_simple_model to true. This will make inverse kinematics at least 10 times faster.

This can be either done fully automatically within Pose2Sim, or manually within OpenSim GUI.

Within Pose2Sim

Scaling and inverse kinematics are performed in a fully automatic way for each trc file.
No need for a static trial!

Model scaling is done according to the mean of the segment lengths, across a subset of frames. We remove the frames where the average knee and hip flexion angles are above 90° (pose estimation is not precise when the person is crouching) and the 50% most extreme segment values (potential outliers).
Note: If the participant is mostly crouching or sitting, manually scale on a standing trial (see next section).

Tip

In your Config.toml file:

Set use_simple_model = true if you want IK to run 10-40 times faster. No muscles, no constraints (eg stiff spine and shoulders, no patella).
Set filter_ik = true if you want to filter angle results after IK with the parameters defined in the [filtering] section. Useful for force estimations if results are noisy.
set use_augmentation = false is you don't want to use the results with augmented marker (this is sometimes better).
Set right_left_symmetry = false if you have good reasons to think the participant is not symmetrical (e.g. if they wear a prosthetic limb).

Open a terminal in your project folder, activate your environment (see here), and run ipython:

from Pose2Sim import Pose2Sim
Pose2Sim.kinematics()

Note

At that point, you should have produced the following files:

Pose2Sim_Project/
├── Config.toml
├── videos/
├── calibration/
│   └── Calib.toml     # Calibration file, either converted or calculated
├── pose/              # 2D pose files
├── pose-sync/         # Synchronized 2D pose files
├── pose-3d/           # 3D marker trajectories, unfiltered and filtered (.trc files)
├── kinematics/        # OpenSim results
│   ├── *_scaled.osim      # Scaled OpenSim model for each person
│   ├── *.mot              # Joint angles for each trial, unfiltered and filtered (open with Excel or│OpenSim)
│   └── opensim_logs.txt
└── logs.txt          # Detailed logs of all the steps

Tip

Visualize your animated skeleton in the Blender add-on or in OpenSim: File -> Preview experimental data.

Once you have the scaled model and the joint angles, you are free to go further!
Ground reaction force estimation, inverse dynamics, muscle analysis, Moco optimization, etc.

Within OpenSim GUI

If you are not fully satisfied with the results or on sitting or crouching trials, you can perform scaling and inverse kinematics in a more traditional way, with (or without) a static trial.

Scaling

Choose a time range where the 3D keypoints are particularly well reconstructed, or capture a static pose, typically an A-pose...
Open OpenSim.
File -> Open Model: Open the provided Model_Pose2Sim_LSTM.osim model from Pose2Sim/OpenSim_Setup.
Note: Here and below, replace 'LSTM' by any other model if needed, e.g. HALPE_26
Tools -> Scale model -> Load: Load the provided Scaling_Setup_Pose2Sim_LSTM.xml scaling file.
Replace the example .trc file with your own data.
Run
File > Save Model: Save the new scaled OpenSim model.

Inverse kinematics

Tools -> Inverse kinematics -> Load: Load the provided IK_Setup_Pose2Sim_LSTM.xml scaling file from Pose2Sim/OpenSim_Setup.
Replace the example .trc file with your own data, and specify the path to your angle kinematics output file.
Run.
Right click on the Model->Motions->Coordinates > Save As: Save angle results

Command line

Alternatively, you can use command-line tools:

import opensim
opensim.ScaleTool("<PATH TO YOUR SCALING OR IK SETUP FILE>.xml").run()
opensim.InverseKinematicsTool("<PATH TO YOUR SCALING OR IK SETUP FILE>.xml").run()

N.B.: You'll need to adjust the time_range, output_motion_file, and enter the absolute path (NOT the relative path) to the input and output .osim, .trc, and .mot files in your setup file.

You can also run other API commands. See there for more instructions on how to use it.

All the parameters

All of the Pose2Sim parameters

Tip

All the parameters are defined and documented in your Config.toml file. You can also pass a dictionary to any Pose2Sim function to override the config parameters (see Demonstration Part-4).

All the parameters (CLICK TO SHOW)

Project

Parameter	Default	Description
`multi_person`	`false`	If `true`, all persons in the scene are analyzed. If `false`, only the person with the lowest reprojection error is kept.
`participant_height`	`'auto'`	Height of the participant(s) in meters. `'auto'`, a float (e.g. `1.72`), or a list of floats (e.g. `[1.72, 1.40]`). Only used for marker augmentation.
`participant_mass`	`70.0`	Mass of the participant(s) in kg. A float or a list of floats. Only used for marker augmentation and scaling; no impact on results unless you need to compute forces.
`frame_rate`	`'auto'`	Frame rate in fps. `'auto'` reads from video metadata, or defaults to 60 fps when working with images.
`frame_range`	`'auto'`	`'auto'`, `'all'`, or a range like `[10, 300]`. `'auto'` trims around frames with low reprojection error. If cameras are not synchronized, designates the frame range of the camera with the shortest recording.
`exclude_from_batch`	`[]`	List of trial paths to exclude from batch analysis, e.g. `['S00_P00_Participant/S00_P00_T00_StaticTrial']`.

Pose

Parameter	Default	Description
`pose_model`	`'Body_with_feet'`	Skeleton model to use. With RTMLib: - `'Body_with_feet'` (HALPE_26, default), - `'Whole_body_wrist'` (COCO_133_WRIST: body+feet+ 2 hand keypoints), - `'Whole_body'` (COCO_133), - `'Lower_body'` (Effectively runs HALPE_26. Later Pose2Sim stages ignore upper-body keypoints, recreate shoulder points: Hip + 0.53 m (Y direction) + 0.1 m (Hip-to-Hip direction), - `'Body'` (COCO_17, Marker augmentation won't work, Kinematic analysis will), - `'Hand'` (Hand_21), - `'Face'` (FACE_106), - `'Animal'` (ANIMAL2D_17). ⚠️ Only RTMPose is natively embeded in Pose2Sim. For all other pose estimation methods, you will have to run them yourself, and then refer to the documentation to convert the output files if needed ⚠️ For Face and Animal, use mode="""{dictionary}""", and find the corresponding .onnx model there https://github.com/open-mmlab/mmpose/tree/main/projects/rtmpose With MMPose: `HALPE_26`, `COCO_133`, `COCO_17`, `CUSTOM`. See CUSTOM example at the end of the Config.toml file. With OpenPose: `BODY_25B`, `BODY_25`, `BODY_135`, `COCO`, `MPII`. With MediaPipe: `BLAZEPOSE`. With AlphaPose: `HALPE_26`, `HALPE_68`, `HALPE_136`, `COCO_133`. With DeepLabCut: `CUSTOM`.
`mode`	`'balanced'`	`'lightweight'`, `'balanced'`, or `'performance'`. Can also be a `"""{dictionary}"""` to manually select detection and pose models (see rtmlib). Models can be local paths or URLs. Make sure `input_sizes` are within square brackets and in the opposite order from the model path (e.g. `[192,256]` for a `256x192` model). If your `pose_model` is not in `skeletons.py`, you may need to create your own. Examples: Equivalent to `mode='balanced'`: `"""{'det_class':'YOLOX', 'det_model':'https://…/yolox_m_8xb8-300e_humanart-c2c7a14a.zip', 'det_input_size':[640,640], 'pose_class':'RTMPose', 'pose_model':'https://…/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.zip', 'pose_input_size':[192,256]}"""` One-stage RTMO (requires `pose_model = 'Body'`): `"""{'pose_class':'RTMO', 'pose_model':'https://…/rtmo-m_16xb16-600e_body7-640x640-39e78cc4_20231211.zip', 'pose_input_size':[640,640]}"""` Animal pose estimation (marker augmentation won't work; custom OpenSim skeleton needed for IK): `"""{'pose_class':'RTMPose', 'pose_model':'https://…/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.zip', 'pose_input_size':[256,256]}"""`
`det_frequency`	`4`	Run person detection only every N frames; bounding boxes are tracked in between. Higher values are faster but may miss detections. Can be as high as you want in simple uncrowded scenes, must be ≥ 1.
`device`	`'auto'`	Inference device: `'auto'`, `'CPU'`, `'CUDA'`, `'MPS'`, or `'ROCM'`.
`backend`	`'auto'`	Inference backend: `'auto'`, `'openvino'`, `'onnxruntime'`, or `'opencv'`.
`parallel_workers_pose`	`'auto'`	`'auto'`, an integer, or `false`. Spawns one worker per video, uses GPU if available. Requires `display_detection = false`, otherwise falls back to sequential.
`display_detection`	`true`	Show real-time pose estimation overlay. Set to `false` for parallel processing.
`overwrite_pose`	`false`	If `false`, skips pose estimation when results already exist.
`save_video`	`'to_video'`	`'to_video'`, `'to_images'`, `'none'`, or `['to_video', 'to_images']`.
`output_format`	`'openpose'`	`'openpose'`, `'mmpose'`, `'deeplabcut'`, `'none'`, or a list. Only `'openpose'` is fully supported downstream.
`average_likelihood_threshold_pose`	`0.5`	Detections are dropped when their average keypoint likelihood is below this threshold.
`tracking_mode`	`'sports2d'`	Person tracker used between frames: `'none'`, `'sports2d'` (fast, recommended), or `'deepsort'` (slower but more customizable).
`max_distance_px`	`100`	Pixels. A person detected farther than this distance from their previous position is treated as a new individual.
`deepsort_params`	`"""{'max_age':30, 'n_init':3, 'nms_max_overlap':0.8, 'max_cosine_distance':0.3, 'nn_budget':200, 'max_iou_distance':0.8}"""`	DeepSort hyperparameters as a `"""{dictionary}"""`. See DeepSort docs for details.
`handle_LR_swap`	`false`	Not implemented yet. Will swap left/right labels if needed.
`undistort_points`	`false`	Not implemented yet. Undistorts 2D points before triangulation.

Synchronization

Parameter	Default	Description
`synchronization_gui`	`true`	If `true`, opens an interactive player to set synchronization parameters manually.
`display_sync_plots`	`true`	Display cross-correlation plots.
`save_sync_plots`	`true`	Save cross-correlation plots to disk.
`keypoints_to_consider`	`'all'`	`'all'` to use every keypoint, or a list such as `['RWrist', 'RElbow']` to focus on points with a clear vertical motion.
`approx_time_maxspeed`	`'auto'`	`'auto'` to search the whole recording (might be slow), or a list of times in seconds (one per camera, eg `[10.0, 2.0, 8.0, 11.0]` ) indicating when the sharpest vertical event occurs.
`time_range_around_maxspeed`	`2.0`	Seconds. Search window is `[approx_time_maxspeed ± time_range_around_maxspeed]`.
`likelihood_threshold_synchronization`	`0.4`	Keypoints with likelihood below this value are ignored during synchronization.
`filter_cutoff`	`6`	Low-pass filter cut-off frequency (Hz) applied before computing cross-correlation.
`filter_order`	`4`	Order of the low-pass filter applied before cross-correlation.

Calibration — Take heart, calibration is not that complicated once you get the hang of it!

Parameter	Default	Description
`calibration_type`	`'convert'`	`'convert'` to import an existing calibration file, or `'calculate'` to compute from scratch.

[calibration.convert]

Parameter	Default	Description
`convert.convert_from`	`'qualisys'`	Source format: `'caliscope'`, `'qualisys'`, `'optitrack'`, `'vicon'`, `'opencap'`, `'easymocap'`, `'biocv'`, `'anipose'`, or `'freemocap'`.
`binning_factor` (qualisys only)	`1`	Usually `1`; set to `2` when filming at 540p with Qualisys.

[calibration.calculate.intrinsics]

Parameter	Default	Description
`overwrite_intrinsics`	`false`	If `false`, skips intrinsic calculation when results already exist.
`intrinsics_extension`	`'jpg'`	File extension of the calibration images or video.
`extract_every_N_sec`	`1`	If a video is provided, extract one frame every N seconds (can be < 1).
`intrinsics_corners_nb`	`[4, 7]`	`[rows, cols]` of internal corners on the checkerboard (one less per side than the printed square count).
`intrinsics_square_size`	`60`	Size of one checkerboard square in mm.
`show_detection_intrinsics`	`true`	Display detected corners during intrinsic calibration.

[calibration.calculate.extrinsics]

Parameter	Default	Description
`calculate_extrinsics`	`true`	Set to `false` to skip extrinsic calculation.
`extrinsics_method`	`'scene'`	`'board'` (checkerboard on floor), `'scene'` (manually clicked points of known 3D coordinates), or `'keypoints'` (coming soon).
`extrinsics_extension`	`'png'`	File extension of the extrinsic calibration image or video.
`show_reprojection_error`	`true`	Display reprojection error after extrinsic calibration.
`moving_cameras`	`false`	Not implemented yet.
`board_position` (board only)	`'vertical'`	`'horizontal'` or `'vertical'`.
`extrinsics_corners_nb` (board only)	`[4, 7]`	`[rows, cols]` of internal corners on the extrinsic checkerboard.
`extrinsics_square_size` (board only)	`60`	Square size in mm (can be `[h, w]` for rectangles).
`object_coords_3d` (scene only)	`[[...], ...]`	List of `[X, Y, Z]` 3D coordinates (in metres) of the points you will click on each camera image. Spread points as widely as possible for best accuracy.

Person Association

Single person

Parameter	Default	Description
`likelihood_threshold_association`	`0.3`	Keypoints with likelihood below this value are ignored.
`reproj_error_threshold_association`	`20`	px. Maximum acceptable reprojection error for a person to be selected.
`tracked_keypoint`	`'Neck'`	Keypoint used to track the person of interest. Choose a stable point visible in all cameras (e.g. `'Neck'`, `'RShoulder'`).

Multi person

Parameter	Default	Description
`reconstruction_error_threshold`	`0.1`	metres. Maximum 3D reconstruction error for two detections to be considered the same person across cameras.
`min_affinity`	`0.2`	Correspondences with affinity below this value are discarded. Affinity is high when reconstruction error ≪ threshold.

Triangulation

Parameter	Default	Description
`reproj_error_threshold_triangulation`	`15`	px. Triangulated points with reprojection error above this threshold are rejected and a camera is removed for retry.
`likelihood_threshold_triangulation`	`0.3`	2D detections with likelihood below this value are ignored for triangulation.
`min_cameras_for_triangulation`	`2`	Minimum number of cameras required to attempt triangulation. Triangulation is skipped for a given frame/keypoint if fewer cameras remain after filtering.
`max_distance_m`	`1.0`	metres. Maximum distance a person can move between frames before being considered a new individual.
`max_unseen_frames`	`100`	Maximum number of consecutive frames a person can be absent before the next detection is assigned a new ID.
`interp_if_gap_smaller_than`	`20`	frames. Gaps smaller than this are interpolated; larger gaps are left as-is (or filled with `fill_large_gaps_with`).
`interpolation`	`'linear'`	Interpolation method for missing points: `'linear'`, `'slinear'`, `'quadratic'`, `'cubic'`, or `'none'`.
`remove_incomplete_frames`	`false`	If `true`, a frame is only kept when all keypoints have been successfully triangulated.
`sections_to_keep`	`'all'`	Which valid sections to retain: `'all'`, `'largest'`, `'first'`, or `'last'`.
`min_chunk_size`	`10`	frames. Minimum length of a consecutive valid section for it to be retained.
`fill_large_gaps_with`	`'last_value'`	How to fill gaps larger than `interp_if_gap_smaller_than`: `'last_value'`, `'nan'`, or `'zeros'`.
`show_interp_indices`	`true`	Print the frame indices that were interpolated for each keypoint.
`make_c3d`	`true`	Also save triangulated data as a `.c3d` file alongside the `.trc` file.

Filtering

Parameter	Default	Description
`reject_outliers`	`true`	Apply a Hampel filter before any other filter. Rejects points outside a 95 % confidence interval from the median in a sliding window of 7 frames. Can be slow on long recordings.
`filter`	`true`	Apply the filter selected by `type` after outlier rejection.
`type`	`'butterworth'`	Filter to apply: `'butterworth'`, `'acc_minimizing'`, `'kalman'`, `'one_euro'`, `'gcv_spline'`, `'gaussian'`, `'LOESS'`, `'median'`, or `'butterworth_on_speed'`.
`display_figures`	`true`	Show filtering results as matplotlib figures.
`save_filt_plots`	`true`	Save filtering plots to disk.
`make_c3d`	`true`	Also save filtered data as a `.c3d` file.

[filtering.butterworth] — most intuitive, standard biomechanics filter

Parameter	Default	Description
`cut_off_frequency`	`6`	3-6 Hz: Walking, slow movements; 6-15 Hz: Running, fast movements; 15+ Hz: Very fast and impulsive movements
`order`	`4`	Filter order.

[filtering.kalman] — Used in countless applications, especially real-time. This simplified Kalman filter assumes constant acceleration with Gaussian process noise

Parameter	Default	Description
`trust_ratio`	`500`	Ratio of measurement trust to process trust (≈ process noise / measurement noise). Higher values follow the data more closely.
`smooth`	`true`	Apply Kalman smoother (non-causal). Set to `false` for true real-time filtering.

[filtering.one_euro] — Simpler and even faster alternative to Kalman filter for real-time, but tends to blunt RoM. Analog to a 1st order Butterworth but with adaptive cut-off frequency. This one is zero-phase

Parameter	Default	Description
`cut_off_frequency`	`4`	Hz. Base cut-off frequency, adapted by velocity.
`beta`	`1.5`	Velocity adaptation coefficient: `f_c = cut_off_frequency + beta × velocity`.
`d_cut_off_frequency`	`1.0`	Hz. Cut-off frequency for the derivative signal.

[filtering.gcv_spline] — Automatically determines optimal parameters for each point, which is good when some move faster than others (eg fingers vs hips)

Parameter	Default	Description
`cut_off_frequency`	`'auto'`	`'auto'` or an integer (behaves like Butterworth). `'auto'` is usually better unless the signal is too short or you got a triangular wave + drift (eg pelvis Y coordinates of a subject walking uphill)
`smoothing_factor`	`1.0`	≥ 0. Values > 1 produce more smoothing; values < 1 follow the data more closely. Ignored when `cut_off_frequency != 'auto'`.

[filtering.acc_minimizing] — Whittaker-Henderson filter (acceleration-minimizing), produces smooth velocities and accelerations, which is important for force estimations but blunts the peaks; recommended for IK output

Parameter	Default	Description
`cut_off_frequency`	`6`	Hz.

[filtering.loess] — Local low-degree polynomial smoothing

Parameter	Default	Description
`nb_values_used`	`5`	Number of neighbouring frames used in the local regression.

[filtering.gaussian] — Weighted moving average, with more weight on central points

Parameter	Default	Description
`sigma_kernel`	`1`	Standard deviation of the Gaussian kernel in frames.

[filtering.median] — Removes outliers (but not as well as the Hampel filter: see reject_outliers)

Parameter	Default	Description
`kernel_size`	`3`	Size of the median filter window in frames (must be odd).

[filtering.butterworth_on_speed] — Signal is differenciated, filtered, then integrated back. Preserves sharp movements, but can introduce an offset in the original signal

Parameter	Default	Description
`cut_off_frequency`	`10`	Hz. Cut-off frequency applied to the velocity signal.
`order`	`4`	Filter order.

marker Augmentation

Requires at least the following markers: [RHip, LHip, RKnee, LKnee, RAnkle, LAnkle, RHeel, LHeel, RSmallToe, LSmallToe, RBigToe, LBigToe]

Parameter	Default	Description
`feet_on_floor`	`false`	If `true`, markers are translated so that the feet touch the floor plane. Useful for ground reaction force or joint load estimation.
`make_c3d`	`true`	Also save augmented marker data as a `.c3d` file.

Kinematics

Parameter	Default	Description
`use_augmentation`	`true`	If `true`, uses the model with augmented markers. Set to `false` if augmentation did not improve results.
`use_simple_model`	`false`	If `true`, uses a simplified OpenSim model (no muscles, no constraints). More than 10× faster; stiff spine and ball-joint shoulders, suitable for most gait tasks.
`filter_ik`	`false`	If `true`, filters joint angle results after IK using the method defined in `[filtering]`. Recommended when force estimation is intended.
`ik_filter_type`	`'acc_minimizing'`	Filter type for IK output. Any type from `[filtering]` is accepted.
`right_left_symmetry`	`true`	Set to `false` if the participant is not bilaterally symmetrical (e.g. if they wear a prosthetic limb).
`default_height`	`1.7`	metres. Fallback height used for model scaling if automatic height estimation fails.
`parallel_workers_kinematics`	`'auto'`	`'auto'`, an integer, or `false`. One worker per person (CPU only). Available in multi-person mode only.
`remove_individual_scaling_setup`	`true`	If `true`, per-person scaling setup XML files are deleted after use to avoid clutter.
`remove_individual_ik_setup`	`true`	If `true`, per-person IK setup XML files are deleted after use to avoid clutter.
`large_hip_knee_angles`	`90`	degrees. Hip and knee angles above this value are considered unreliable and excluded from scaling.
`trimmed_extrema_percent`	`50`	Percentage of the most extreme segment-length values removed before computing the mean for scaling.

Logging

Parameter	Default	Description
`use_custom_logging`	`false`	Set to `true` when Pose2Sim is embedded in an application that already configures Python logging.

Utilities

A list of useful standalone tools

We provide a list of standalone tools (see Utilities), which can be either run as scripts, or imported as functions. Check usage in the docstring of each Python file. The figure below shows how some of these tools can be used to further extend Pose2Sim usage.

Open a terminal in your project folder, activate your environment (see here), and try any of these. Type in name_of_script.py -h for more instructions on how to use them.

Video editing (CLICK TO SHOW)

face_blurring.py Blurs or masks faces on a video.

Converting pose files (CLICK TO SHOW)

Blazepose_runsave.py
Runs BlazePose on a video, and saves coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format.
DLC_to_OpenPose.py
Converts a DeepLabCut (h5) 2D pose estimation file into OpenPose (json) files.
AlphaPose_to_OpenPose.py
Converts AlphaPose single json file to OpenPose frame-by-frame files.

Converting calibration files (CLICK TO SHOW)

calib_toml_to_easymocap.py Converts an OpenCV .toml calibration file to EasyMocap intrinsic and extrinsic .yml calibration files.

calib_easymocap_to_toml.py Converts EasyMocap intrinsic and extrinsic .yml calibration files to an OpenCV .toml calibration file.

calib_from_checkerboard.py Calibrates cameras with images or a video of a checkerboard, saves calibration in a Pose2Sim .toml calibration file. You should probably use Pose2Sim.calibration() instead, which is much easier and better.

calib_qca_to_toml.py Converts a Qualisys .qca.txt calibration file to the Pose2Sim .toml calibration file (similar to what is used in AniPose).

calib_toml_to_qca.py Converts a Pose2Sim .toml calibration file (e.g., from a checkerboard) to a Qualisys .qca.txt calibration file.

calib_toml_to_opencap.py Converts an OpenCV .toml calibration file to OpenCap .pickle calibration files.

calib_toml_to_opencap.py To convert OpenCap calibration tiles to a .toml file, please use Pose2Sim.calibration() and set convert_from = 'opencap' in Config.toml.

Plotting tools (CLICK TO SHOW)

json_display_with_img.py Overlays 2D detected json coordinates on original raw images. High confidence keypoints are green, low confidence ones are red.

json_display_without_img.py Plots an animation of 2D detected json coordinates.

trc_plot.py Displays X, Y, Z coordinates of each 3D keypoint of a TRC file in a different matplotlib tab.

Other trc tools (CLICK TO SHOW)

trc_from_easymocap.py Convert EasyMocap results keypoints3d .json files to .trc.

c3d_to_trc.py Converts 3D point data from a .c3d file to a .trc file compatible with OpenSim. No analog data (force plates, emg) nor computed data (angles, powers, etc.) are retrieved.

trc_to_c3d.py Converts 3D point data from a .trc file to a .c3d file compatible with Visual3D.

trc_desample.py Undersamples a trc file.

trc_scale.py Scale trc coordinates by a desired factor.

trc_rotate.py Rotate trc coordinates by 90° around an axis. You can either choose an axis to rotate around, or use one of the predefined conversions from and axis-up to another one.

trc_Zup_to_Yup.py Changes Z-up system coordinates to Y-up system coordinates.

trc_mot_filter.py Filters trc or mot files. Available filters: Butterworth, Kalman, Butterworth on speed, Gaussian, LOESS, Median.

trc_gaitevents.py Detects gait events from point coordinates according to Zeni et al. (2008).

trc_combine.py Combine two trc files, for example a triangulated DeepLabCut trc file and a triangulated OpenPose trc file.

trc_from_mot_osim.py Build a trc file from a .mot motion file and a .osim model file.

bodykin_from_mot_osim.py Converts a mot file to a .csv file with rotation and orientation of all segments.

reproj_from_trc_calib.py Reprojects 3D coordinates of a trc file to the image planes defined by a calibration file. Output in OpenPose or DeepLabCut format.

How to cite and how to contribute

How to cite

If you use this code or data, please cite Pagnon et al., 2022b, Pagnon et al., 2022a, or Pagnon et al., 2021.

@Article{Pagnon_2022_JOSS, 
  AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel}, 
  TITLE = {Pose2Sim: An open-source Python package for multiview markerless kinematics}, 
  JOURNAL = {Journal of Open Source Software}, 
  YEAR = {2022},
  DOI = {10.21105/joss.04362}, 
  URL = {https://joss.theoj.org/papers/10.21105/joss.04362}
 }

@Article{Pagnon_2022_Accuracy,
  AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
  TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy},
  JOURNAL = {Sensors},
  YEAR = {2022},
  DOI = {10.3390/s22072712},
  URL = {https://www.mdpi.com/1424-8220/22/7/2712}
}

@Article{Pagnon_2021_Robustness,
  AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
  TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 1: Robustness},
  JOURNAL = {Sensors},
  YEAR = {2021},
  DOI = {10.3390/s21196530},
  URL = {https://www.mdpi.com/1424-8220/21/19/6530}
}

How to contribute and to-do list

Tip

If you want to contribute to Sports2D or Pose2Sim, please see this issue or join the Discord community!

Pose2Sim releases (details here):

Main to-do list:

Graphical User Interface
Self-calibration based on keypoint detection + Bundle adjustment + Calibration of moving cameras
Get rid of the brute-force single-person mode, and instead automatically or manually select the persons of interest in multi-person mode.
Real-time processing 3D kinematics
Support ground reaction force, joint moments, muscle activation estimation
3D monocular solution

Detailed GOT-DONE and TO-DO list (CLICK TO SHOW)

✔ Pose: Support OpenPose body_25b for more accuracy, body_135 for pronation/supination.

✔ Pose: Support BlazePose for faster inference (on mobile device).

✔ Pose: Support DeepLabCut for training on custom datasets.

✔ Pose: Support AlphaPose as an alternative to OpenPose.

✔ Pose: Define custom model in config.toml rather than in skeletons.py.

✔ Pose: Integrate pose estimation within Pose2Sim (via RTMlib).

▢ Pose: Support MMPose, SLEAP, etc.

▢ Pose: Optionally let user select the person of interest in single_person mode: multiperson = true # true, or 'single_auto', or 'single_click'. 'single_auto' selects the person with lowest reprojection error, and 'single_click' lets the user manually select the person of interest.

▢ Pose: Implement RTMPoseW3D and monocular 3D kinematics

▢ Pose: Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...)

▢ Pose: GUI help for DeepLabCut model creation.

✔ Calibration: Convert Qualisys .qca.txt calibration file.

✔ Calibration: Convert Optitrack extrinsic calibration file.

✔ Calibration: Convert Vicon .xcp calibration file.

✔ Calibration: Convert OpenCap .pickle calibration files.

✔ Calibration: Convert EasyMocap .yml calibration files.

✔ Calibration: Convert bioCV calibration files.

✔ Calibration: Easier and clearer calibration procedure: separate intrinsic and extrinsic parameter calculation, edit corner detection if some are wrongly detected (or not visible).

✔ Calibration: Possibility to evaluate extrinsic parameters from cues on scene.

▢ Calibration: Automatic calibration based on keypoints. Set world reference frame in the end.

▢ Calibration: Calibration of moving cameras. Detect or click points, and then track them for live calibration of moving cameras. Propose to click again when they are lost. Alternatively, use DVPO (see its implementation in GVHMR)

▢ Calibration: Support vertical checkerboard.

▢ Calibration: Calibrate cameras by pairs and compute average extrinsic calibration with aniposelib.

▢ Calibration: Fine-tune calibration with bundle adjustment.

▢ Calibration: Support ChArUco board detection (see there).

▢ Calibration: Calculate calibration with points rather than board. (1) SBA calibration with wand (cf Argus, see converter here). Set world reference frame in the end.

▢ Calibration: Convert fSpy calibration based on vanishing point.

✔ Synchronization: Synchronize cameras on keypoint speeds.

✔ Synchronization: Synchronize in multi-person mode: click on the person to synchronize on.

✔ Person Association: Automatically choose the main person to triangulate.

✔ Person Association: Multiple persons association. 1. Triangulate all the persons whose reprojection error is below a certain threshold (instead of only the one with minimum error), and then track in time with speed cf Slembrouck 2020? or 2. Based on affinity matrices Dong 2021? or 3. Based on occupancy maps Yildiz 2012? or 4. With a neural network Huang 2023?

✔ Triangulation: Triangulation weighted with confidence.

✔ Triangulation: Set a likelihood threshold below which a camera should not be used, a reprojection error threshold, and a minimum number of remaining cameras below which triangulation is skipped for this frame.

✔ Triangulation: Interpolate missing frames (cubic, bezier, linear, slinear, quadratic)

✔ Triangulation: Show mean reprojection error in px and in mm for each keypoint.

✔ Triangulation: Show how many cameras on average had to be excluded for each keypoint.

✔ Triangulation: Evaluate which cameras were the least reliable.

✔ Triangulation: Show which frames had to be interpolated for each keypoint.

✔ Triangulation: Solve limb swapping (although not really an issue with Body_25b). Try triangulating with opposite side if reprojection error too large. Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line.

✔ Triangulation: Undistort 2D points before triangulating (and distort them before computing reprojection error).

✔ Triangulation: Offer the possibility to augment the triangulated data with the OpenCap LSTM. Create "BODY_25_AUGMENTED" model, Scaling_setup, IK_Setup.

✔ Triangulation: Multiple person kinematics (output multiple .trc coordinates files). Triangulate all persons with reprojection error above threshold, and identify them by minimizing their displacement across frames.

▢ Triangulation: Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution.

▢ Triangulation: Offer the possibility of triangulating with Sparse Bundle Adjustment (SBA), Extended Kalman Filter (EKF), Full Trajectory Estimation (FTE) (see AcinoSet).

▢ Triangulation: Implement normalized DLT and RANSAC triangulation, Outlier rejection (sliding z-score?), as well as a triangulation refinement step.

▢ Triangulation: Track hands and face, and add articulated OpenSim hand.

✔ Filtering: Available filtering methods: Butterworth, Butterworth on speed, Gaussian, Median, LOESS (polynomial smoothing).

✔ Filtering: Implement Kalman filter and Kalman smoother.

▢ Filtering: Implement smoothNet

✔ OpenSim: Integrate better spine from lifting fullbody model to the gait full-body model, more accurate for the knee.

✔ OpenSim: Optimize model marker positions as compared to ground-truth marker-based positions.

✔ OpenSim: Add scaling and inverse kinematics setup files.

✔ OpenSim: Add full model with contact spheres (SmoothSphereHalfSpaceForce) and full-body muscles (DeGrooteFregly2016Muscle), for Moco for example.

✔ OpenSim: Add model with ISB shoulder.

✔ OpenSim: Integrate OpenSim in Pose2Sim.

✔ OpenSim: Do not require a separate scaling trial: scale on the 10% slowest frames of the moving trial instead, or take median scaling value.

▢ OpenSim: Implement optimal fixed-interval Kalman smoothing for inverse kinematics (this OpenSim fork), or Biorbd)

✔ GUI: Blender add-on (cf MPP2SOS), Maya-Mocap and BlendOsim.

▢ GUI: Pose2Sim_Blender: Rig skeleton (see Caliscope)

▢ GUI: App or webapp (e.g., with gradio, Streamlit, or Napari ). Also see tkinter interfaces (or Kivy if we want something nice and portable, or Google Colab). Maybe have a look at the Deeplabcut GUI for inspiration.

▢ GUI: 3D plot of cameras and of triangulated keypoints.

▢ GUI: Demo on Google Colab (see Sports2D for OpenPose and Python package installation on Google Drive).

✔ Demo: Provide Demo data for users to test the code.

✔ Demo: Add videos for users to experiment with other pose detection frameworks

✔ Demo: Time shift videos and json to demonstrate synchronization

✔ Demo: Add another virtual person to demonstrate personAssociation

▢ Tutorials: Make video tutorials.

▢ Doc: Use Sphinx, MkDocs, or github.io (maybe better) for clearer documentation.

✔ Pip package

✔ Batch processing (also enable non-batch processing)

✔ Catch errors

✔ Integrate Sports2D for OpenSim analysis from a single camera

▢ Conda package

▢ Docker image

▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames)

▢ Config parameter for non batch processing

▢ Run from command line via click or typer

▢ Utilities: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)

▢ Utilities: Create trc_to_c3d.py script

✔ Bug: calibration.py. FFMPEG error message when calibration files are images. See there.

✔ Bug: common.py, class plotWindow(). Python crashes after a few runs of Pose2Sim.filtering() when display_figures=true. See there.

Acknowledgements:

Supervised my PhD: @lreveret (INRIA, Université Grenoble Alpes), and @mdomalai (Université de Poitiers).
Post-doc at the University of Bath (UK) and engineer at ForceTeck.
Demo data from @aaiaueil from Université Gustave Eiffel.
Thanks to all the contributors, past and to come! Thanks to everyone who gave feedback and contributed to making this software program better.

Name		Name	Last commit message	Last commit date
Latest commit History 1,260 Commits
.github/workflows		.github/workflows
Content		Content
Pose2Sim		Pose2Sim
.gitignore		.gitignore
CITATION.cff		CITATION.cff
LICENSE		LICENSE
README.md		README.md
pose2sim.yaml		pose2sim.yaml
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

Pose2Sim

Contents

Installation and Demonstration

Installation

1. Set up a uv environment:

2. Install Pose2Sim:

3. Optional:

Demonstration Part-1: End to end video to 3D joint angle computation

Demonstration Part-2: Visualize your results with OpenSim or Blender

Basic visualization with the OpenSim GUI

Further investigation with the Pose2Sim Blender add-on

Demonstration Part-3: Try multi-person and batch analyses

Multi person analysis

Batch processing

Demonstration Part-4: Go further

Too slow for you?

Use on your own data

Setting up your project

2D pose estimation

With RTMPose (default):

With MMPose (coming soon):

With DeepLabCut:

With OpenPose (legacy):

With MediaPipe BlazePose (legacy):

With AlphaPose (legacy):

Camera calibration

Convert from Caliscope, AniPose, FreeMocap, Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV

Calculate from scratch

1. Calculate intrinsic parameters with a checkerboard

2.Calculate extrinsic parameters

Synchronization

Associate persons across cameras

Triangulating keypoints

Filtering 3D coordinates

Marker Augmentation

OpenSim kinematics

Within Pose2Sim

Within OpenSim GUI

Command line

All the parameters

Utilities

How to cite and how to contribute

How to cite

How to contribute and to-do list

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 81

Uh oh!

Contributors

Uh oh!

Languages