Skip to content

Home

Jump to bottom
Alexander Refsum Jensenius edited this page Apr 14, 2026 · 14 revisions

The Musical Gestures Toolbox for Python is a collection of high-level modules targeted at researchers working with video recordings. It includes visualization techniques such as motion videos, motion history images, and motiongrams; techniques that, in different ways, allow for looking at video recordings from different temporal and spatial perspectives. It also includes basic computer vision analysis, such as extracting the quantity and centroid of motion, and using such features in analysis.

The toolbox was initially developed to analyze music-related body motion (of musicians, dancers, and perceivers) but is equally helpful for other disciplines working with video recordings of humans, such as linguistics, pedagogy, psychology, and medicine.

Functionality

The Musical Gestures Toolbox contains functions to analyze and visualize video, audio, and motion capture data. There are three categories of functions:

  • Preprocessing (trimming, cropping, color adjustments, etc.)
  • Visualization (video playback, image display, plotting)
  • Processing (videograms, average images, motion images, etc.)

Getting Started Options

  1. Quick Start Tutorial
  2. Jupyter Notebook – can also be run in Colab
  3. Browse the detailed function documentation in this wiki
  4. Examples Overview
  5. Complete Documentation Site

Recent Updates (April 2026)

  • CI and packaging reliability: Fixed modern setuptools compatibility by using a valid PEP 517 backend (setuptools.build_meta) and restoring a minimal setup.py shim (setup()) for metadata generation.
  • GitHub Actions modernization: Updated CI workflows to current action versions and removed a deprecated Node 20-based FFmpeg setup action from the Windows job.
  • GPU acceleration controls: Added optional CUDA paths for optical flow and face blurring with robust CPU fallback, and exposed get_cuda_device_count() to check OpenCV CUDA visibility.
  • Safer defaults for portability: flow.dense(), flow.sparse(), and blur_faces() now default to CPU (use_gpu=False), with explicit GPU opt-in.
  • Documentation alignment: Updated README/docs/API pages to reflect new GPU defaults and usage patterns, and aligned the docs citation with the SMC 2021 paper reference.
  • Test stability improvements: Added GPU-path tests and made ffprobe-dependent utility assertions more tolerant across platforms where frame/duration rounding differs slightly.
  • Notebook verification: The main toolbox notebook was executed with extended timeout and allow-errors mode to validate end-to-end execution in this environment.

Wiki Navigation

This wiki provides detailed documentation for individual MGT functions:

Backend

The speed and efficiency of the MGT are made possible by the excellent FFmpeg project. Many of the toolbox functions are Python wrappers/bindings on FFmpeg commands called in a subprocess.

Problems

Please help improve the toolbox by adding bugs and feature requests in the issues section.

A project from the fourMs Lab, RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, Department of Musicology, University of Oslo.

Clone this wiki locally