Write a python program that will analyze the two dashcam videos provided (mcgill_drive.mp4 and st-catherines_drive.mp4, each are 30 frames per second, and are taken with the same car/dashcam) and provide the following analytics:
- Number of parked cars passed
- Number of moving cars passed
- Number of pedestrians passed
- Bonus output: Maximum speed in km/hour of the car with the dashcam You can use any software (except for software developed by other students in the class).
Write a report that provides the following:
- detailed description of the overall approach taken. State clearly any assumptions that you made.
- descriptions of each software package or routine used
- summary of program output on the two videos, with comparison to manually obtained ground truth values
- discussion of program performance and problems
- all python code that you developed (best done by submitting the report as a Jupyter notebook with embedded code) In doing this assignment, it is best to think like an engineer - think about what information is needed to provide the required results, and how do we get this information? The needed information is not always to be found in the image data.
This assignment can be done in pairs or individually. If done in pairs, only one report need be submitted, just remember to clearly indicate the names and student numbers of each person in the group.
| Video Name | Pedestrians | Total Cars | Parked Cars | Driving Cars |
|---|---|---|---|---|
| mcgill_drive.mp4 | 43 | 45 | 17 | 28 |
| st-catherines_drive.mp4 | 120 | 70 | 60 | 10 |
| Video Name | Pedestrians | Total Cars | Parked Cars | Driving Cars | Max Speed | Average Speed |
|---|---|---|---|---|---|---|
| mcgill_drive.mp4 | 44 / 43 | 45 / 43 | 16 / 17 | 29 / 28 | 39.01 km/h | 24.42 km/h |
| St-catherines_drive.mp4 | 114 / 120 | 69 / 70 | 60 / 60 | 9 / 10 | 59.53 km/h | 30.71 km/h |
McGill Drive dashcam results
St-Catherines Drive dashcam results
When measuring ground truth values:
- Bikers are not pedestrians
- Trucks, busses and motorcycles are not counted as cars
- Cars that are idle but not parked count as moving cars (a car at an intersection)
- Break the video up into individual frames and run a callback function on every frame to do the detection, tracking and annotation
- Objects are assigned a unique ID and added to a set to avoid double counting
- Overlay is updated according to the length of these unique object lists (parked cars, moving cars and pedestrians)
- Trackers are used to keep track of unique objects
- Retain a tracker_id through occlusions of up to 100 frames
- Require a minimum inference to be tracked
- Optical Flow used to measure speed by getting displacement of objects measured through multiple frames
- Parked cars determined as cars that have displacement less than a certain threshold
- Moving cars are cars that have displacement over a certain threshold or are within a polygon in the street
- Pedestrians are people who are walking on foot
- Bikers and cyclists were filtered out by checking to see if a person's boundary box intersects with a bicycle or motorcycle boundary box
- Object detection model used was ultralytics YOLOv8x
- The library used for object tracking, object annotations, object detection and video processing was supervision
- This library breaks down a video and allows you to run a callback function on each frame to process a video
- This library has built in functionality for object detection using YOLOv8x, object tracking and annotations over the original video
- The library used for the overlay creation and optical flow implementation was cv2
- cv2 has an optical flow implementation, we used the function calcOpticalFlowFarneback() which gave us an estimated speed for different parts of the video
- cv2 was used to design a simple overlay to display current object counts and speed using the method putText()
Summary of program output on the two videos, with comparison to manually obtained ground truth values:
- The results for the video can be seen live on the overlay while watching the result video
- A summary of the results can also be seen in the output of our code
- Speed estimation is not very precise and fluctuates a lot
- Some pedestrians are not detected because of poor lighting conditions
- Reflections cause double counting for some cars and pedestrians due to two objects being detected
- Cars that are stopped at intersections are sometimes detected as parked
- Some mailboxes are mistaken for people
First to get an idea to see if our results were right we wanted to get an estimate of the speed based on the distance (aquired from google maps) and the duration of the drive (from video):
-
McGill drive video:
- Measured distance on google map: 300m
- Duration of drive: 35s
- → Average speed = 300/35 = 8.5 m/s = 30.6 km/h
-
St Catherine drive video:
- Measured distance on google map: 430m
- Duration of drive: 42s (actually 49s but slows down at red light at the end)
- → Average speed = 430/42 = 10.23 m/s = 36.72 km/h
We then used optical flow to estimate the speed of the car in the video
- The speed estimation was not very precise and fluctuated a lot
- To correct for this we "smoothed out" the speed estimation by taking the average speed and adjusting the differences from the average speed using a sigmoid function.
- We also got a speed value for every frame and then averaged the speed over 30 frames (1 second since video is 30 fps) to get a more accurate speed estimation
Note that estimating the speed for both videos takes a significant amount of time, so we implemented a way to save the speeds to a file and load them in the future to avoid recalculating them.