WhaleMoanViz

Michaela Noel Alksne and Shane Andres

WhaleMoanViz (WMV) is a Triton Remora designed for visualizing, editing, and validating automated whale call detections generated by WhaleMoanDetector. WMV is capable of creating new training examples for WhaleMoanDetector, allowing for incremental improvements of a model from human feedback (active learning).

This project is still in development, so please reach out with any issues or feature requests you might have!

Getting Started

1. WMV requires an installation of Triton, version 1.02 or higher.

2. Download WMV through GitHub:

   • Click the green "Code" button at the top of this page
   • Select "Download ZIP"
   • Extract the ZIP file to a location you'll remember

   Alternatively, if you have git installed run the following command:

   git clone https://github.com/m1alksne/WhaleMoanViz.git

3. Add WMV to Triton:

   • In the Triton control window, select "Remoras"->"Add Remora" from the menu
   • Navigate to the WhaleMoanViz folder and add it

4. If using WhaleMoanDetector for active learning, configure your project settings:

   • Navigate to the location of your Triton installation
   • Open Triton/Remoras/WhaleMoanViz/config.m and edit:

   REMORA.lt.lVis_det.labels = {'A', 'B', 'D', '20Hz', '40Hz'};                % Call types
   REMORA.lt.lVis_det.wmdFolder = 'C:/path/to/WhaleMoanDetector';              % Where your WhaleMoanDetector folder is
   REMORA.lt.lVis_det.pyenvFolder = 'C:/path/to/your/python/environment';      % Where your whalemoandetector python environment is

5. Open Triton and launch WMV from the Control window by selecting "Remoras"->"WhaleMoanViz"->"Visualize Labels" or "Create Labels".

Editing Detections

When creating new annotations or visualizing predicted detections, the WMV window appears as:

1. Load button - used to import predicted bounding boxes from a WhaleMoanDetector model's output.
2. Name of the file containing the predicted detections currently being shown. The checkbox can be used to toggle between showing and hiding the predictions.
3. Add detection button - used to add a detection that the model missed when reviewing predicted bounding boxes.
4. Previous / next detection button - used to move the current plot to show the previous or next detection.
5. Previous / next section button - used to navigate backwards or forwards in the current plot.
6. Previous / next file button - used to load audio files from the folder the current audio file is in.
7. Save edits button - used to save the current working version of the annotations.
8. Make new examples button - used to create new training examples (annotations and spectrograms) for training a WhaleMoanDetector model.

Loading Detections

1. Use the Triton control window to load in one of the audio files in your dataset. Change the spectrogram settings as desired.

2. Click the load button in the WMV window, and load in your predicted detections. These should be in the model output format (see Appendix).

Bounding boxes will appear around any detections in the current window. The following sections describe how to work with these detections.

Selecting a Detection

To select a detection, click anywhere inside of the bounding box. The bounding box should turn blue:

Click and drag the handles to change the size of the box. To confirm changes, press enter. To deselect, press the escape key or click somewhere else in the spectrogram.

Creating a Detection

To mark a call missed by the model (a false negative), click the "Add Detection" button. Click and drag on the spectrogram to create a new bounding box. When finished, the box will turn green:

To delete the bounding box, right click on the box and select "Delete". Alternatively, you can select the box and press the backspace or delete key. To change its label, right click on the box and select "Change Label".

Marking a False Detection

To mark a detection as incorrect (a false positive), right click on the box and select "Mark FP". Alternatively, you can select the box and press the backspace or delete key. The bounding box will turn red:

To undo (mark the box as a true positive), right click on the box and select "Mark TP".

Saving Detections

Saving Edits

When you are satisfied with your evaluation of the detector, you can save the edited detections by clicking “Save Edits.” This will save a new .txt file to the same directory. The file name will be the same as that of the detections, but with the word "raw" replaced with "verified" (ex. raw_detections.txt -> verified_detections.txt).

This file will also be stored in the model output format (see Appendix) with an extra column called “pr”. This notes whether you marked each detection as true or false with the following numbering scheme:

True positive = 1

False positive = 2

False negative = 3

Making New Examples

To create new training examples for a WMD model with the edited detections, click the "Make New Examples" button. Once you select a location to save the new examples to, a command prompt window should appear with a progress bar:

The process will:

1. Display progress with a real-time progress bar showing the current audio file being processed
2. Create a file new_examples.txt containing all detections marked as true positive or false negative, in the model input format (see Appendix)
3. Save each spectrogram containing a detection to the spectrogram_folder specified in the WMD config.yaml file

Troubleshooting

1. Suddenly unable to see detections in current window

• Try deselecting and reselecting the checkbox in the WMV window
• Use the WMV buttons to navigate through audio instead of Triton's - using Triton's will not prompt the plot window to draw bounding boxes.

2. Accidentally closed WMV while editing detections

• There are safeguards in place to prevent this, but if it occurs there is still hope!
• Your detections are still stored in the Triton REMORA global variable. Reference the Remora's code for information on how to access this

Appendix

Model Input Format

Training annotations should be tab-delimited text files with the following columns:

Column	Description	Example
spectrogram_path	Path to the spectrogram image	C:/spectrograms/audio1_20230801T120000.png
label	Call type	A, B, D, 20Hz, 40Hz
xmin, xmax, ymin, ymax	Bounding box coordinates in the spectrogram image	150, 250, 80, 120

Rows with only spectrogram_path filled represent hard negative examples (spectrograms with no calls).

Model Output Format

Prediction files are tab-delimited with the following columns:

Column	Description	Example
wav_file_path	Source audio file path	C:/audio/audio1.wav
model_no	Name of the model that made the detection	my_model
image_file_path	Path to the spectrogram image	C:/spectrograms/audio1_20230801T120000.png
label	Detected call type	A, B, D, 20Hz, 40Hz
score	Confidence score (0-1, higher is more confident)	0.50
start_time_sec, end_time_sec	Detection start / end time in seconds from file start	15, 34
start_time, end_time	Absolute start / end timestamp	2023-08-01 12:00:15, 2023-08-01 12:00:34
min_frequency, max_frequency	Minimum / maximum frequency of the detection (Hz)	23, 40
box_x1, box_x2, box_y1, box_y2	Bounding box coordinates in the spectrogram image	169.889, 263.782, 63.582, 77.574