This repository contains a series of scripts that can be used to track the positions of fireflies in 3D space, given two videos of the same swarm of fireflies taken from different angles. The following sections detail the requirements for running this workflow and provide instructions for using it.
The detection script is written in C++, and the accompanying scripts are written in Python and MATLAB. The tools can be run on any platform, provided that the following dependencies are installed.
CMake is used for coordinating cross-platform compilation of the C++ source code. Downloads for each operating system can be found at the following link: https://cmake.org/download/. Version 3.31 should be used for most likely compatibility with the provided source code and build files.
On Windows, you will also need MSVC, which can be found at the following link: https://visualstudio.microsoft.com/vs/features/cplusplus/. Unfortunately there is not currently a way to install the compiler without installing the Visual Studio application.
On Mac and Linux, you will also need G++, which should be pre-installed.
The Python runtime environment is used to run the analysis scripts. Downloads for each operating system can be found at the following link: https://www.python.org/downloads/. Version 3.12 should be used for most likely compatibility with the provided source code.
OpenCV is used to display a live preview of the input video with a tracking overlay. Downloads for each operating system can be found at the following link: https://opencv.org/releases/. Version 4.11 should be used for most likely compatibility with the provided source code.
FFmpeg is used to read data from the input video and send it in an appropriate format to the tracking script. Downloads for each operating system can be found at the following link: https://www.ffmpeg.org/download.html.
MATLAB is used to calibrate the two cameras and triangulate the firefly positions in 3D space. Version R2024b should be used for most likely compatibility with the provided source code.
The C++ code can be compiled with the following commands, beginning in the directory that contains CMakeLists.txt.
mkdir build
cmake -B build
cmake --build build --config ReleaseOn Windows, these commands will place the final executable in the directory build\Release. On Mac and Linux, these commands will place the final executable in the directory build.
The entire workflow can be broken up into the following steps, each of which must be run separately:
- Determine temporal alignment of videos
- Determine physical alignment of cameras
- Track fireflies in each video
- Optimize tracked data
- Match pairs of tracked fireflies
- Triangulate 3D positions over time
Ensure that all generated output files are placed into the same directory.
To find the temporal alignment of the two videos, run the following Python command:
python scripts/align_audio.py <left_video> <right_video>To find the physical alignment (stereo parameters) of the two cameras, run the following command in MATLAB from the scripts directory:
params <left_photo_dir> <right_photo_dir>Tracking the fireflies in 2D/camera space is done separately for each video, using the temporal offset obtained in the first step. If the offset is positive, use the offset for the right video and 0 for the left video. If the offset is negative, use the positive version of the offset for the left video and 0 for the right video.
On Windows, use the following command:
.\run <video_path> <offset>On Mac and Linux, use the following command:
./run.sh <video_path> <offset>The data from each video will be saved in a file called out.csv. Make sure to rename the output files to left.csv and right.csv before continuing. Also make sure to change the name of the first output file before beginning to track the second video, or it will be overwritten.
Sometimes the firefly series can become segmented during the initial tracking, but they can often be stitched back together after the fact. To do this, run the following Python command for each tracked script:
python scripts/cull.py <data_path>This will save a copy of the provided output file with the combined series.
To pair up the firefly IDs between the two videos, run the following Python command:
python scripts/match.py <data_dir>To triangulate the 3D positions of the matched pairs, run the following command in MATLAB:
generateSeries <data_dir>All the source code included publicly in this repository was written by Isaac Brown and Bowen Cheng, under the supervision of Guy Amichay and Daniel Abrams at Northwestern University. The project uses OpenCV for retrieving camera input and creating a GUI preview, although the source code for OpenCV is not directly included in this repository.