FSBRT Camera Pipeline Repository

This repository contains all necessary Camera and Neural Network files for FSB Racing Team (Autonomous division). Each individual package contained, has its own README.md file but here you can find all needed instructions combined to run all packages.

For more detailed look into this pipeline take a look at the pdf report.

Table of Contents

• Pipeline
  • Hardware and software explained
  • Installation
• Realsense Camera driver setup
  • FRAMOS CameraSuite Package Installation
  • Camera IP Address configuration
  • ROS wrapper
• YOLOv7 Neural Network setup
  • ROS Node Setup & Usage
• 2D pixel to 3D point Deprojection

Hardware and software

The current hardware setup used in the car (RT06D Vulpes) includes a FRAMOS Realsense D435e Stereo camera mounted above the driver's head.
The software needed to run the camera and access its streams through ROS (framos_driver), run cone object detection (neural_network/yolov7-ros) and obtain the 3D position of those cones (neural_network/2D3D_deprojection) are all contained in this repository.

Prerequisites

• Ubuntu 20.04 x86_64
• ROS Noetic
• Python 3.8 or higher

Installation

First, clone the repo into your catkin workspace and build the package:

cd ~/catkin_ws/src
git clone https://github.com/marinmaletic/fsbrt-camera.git
cd ..
sudo apt install ros-noetic-ddynamic-reconfigure
catkin build # if the build is giving errors, follow camera driver setup and repeat the build

The Python requirements are listed in the requirements.txt. You can simply install them as

cd fsbrt-camera
pip install -r requirements.txt

Realsense Camera driver setup

In the framos_driver folder, a ROS wrapper is available. This section will lead you through the installation of the Software Package required for the wrapper and operating the Realsense Camera.

Requirements

• FRAMOS CameraSuite version 4.10.0.0 or higher (installed in next section)
• Intel® RealSense™ SDK with support for D400e camera series version 2.50.10 or higher (installed in next section)
• Make sure that main, universe, restricted and multiverse repositories are added:

sudo add-apt-repository main
sudo add-apt-repository universe
sudo add-apt-repository restricted
sudo add-apt-repository multiverse
sudo apt update

FRAMOS CameraSuite Package Installation

For the ROS driver to work, appropriate Software Package is needed to be installed. The FRAMOS CameraSuite Package is available for download here. Unzip the file, position yourself in it, and follow these installation instructions:

sudo apt-get update
sudo apt-get install linux-headers-$(uname -r)

Install FRAMOS CameraSuite - during installation, you can choose to install a filter driver (recommended) and add current user to 'video' group when prompted:

sudo apt install ./FRAMOS_CameraSuite*.deb

Reboot the system to finish the installation and next install Intel® RealSense™ SDK with support for D400e cameras:

sudo apt install ./FRAMOS-librealsense2*.deb

To verify the installation, launch Intel® RealSense™ Viewer from terminal:

realsense-viewer

Camera IP Address configuration

In order for the computer to recognize the camera, its IP address must be configured. Before start, perform the installation of the Framos Software Package (CameraSuite and Librealsense2). To validate that the camera is recognized properly please check using the ConfigureIP tool from CameraSuite in the /usr/src/framos/camerasuite/Tools directory:

If camera does not appear connected, please review the network settings on host PC and Camera. Using ConfigureIP tool, we can set persistent or temporary (resets when camera shutsdown) IP Address for the camera. It is important that the camera's IP address is in the same subnet as the computer (or router). At the moment, in our setup, the computer has a subnet of 192.168.1.x, and the router 192.168.140.x.
Example: When the camera is connected to the PC over the router, the IP Address of the Camera must be set to e.g. 192.168.140.5 vs 192.168.1.5 when connected directly to the PC.
Note: If the Camera isn't connecting on the Host PC network settings, manually set the IP Address in these settings to a different IP Address then what is the camera's internal IP Address.

Refer to the troubleshooting section on the FRAMOS website or the pdf report as well.

ROS Wrapper Setup & Usage

After having built the packages using catkin build, source the devel/setup.bash script and launch the camera node:

roslaunch realsense2_camera framos_rs_camera.launch

A number of camera parameters can be changed using a configuration file located in framos_driver/realsense2_camera/config/config_file.yaml :

• RGB and Depth resolutions
• RGB and Depth FPS
• Allignment of depth to color (True of False)
• Packet sizes and Inter packet delay

YOLOv7 Neural Network setup

This YOLOv7 version is a fork of Lukas Ewecker's ROS package with the necessary changes for our implementation for cone detection. It wraps the official implementation into a ROS node (so most credit goes to the YOLOv7 creators).

Note: There are currently two YOLOv7 variants out there. This repo contains the implementation from the paper YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.

Custom training weights

The YOLOv7 was trained on the FSOCO Dataset. Download the custom weights here and paste the unzipped .pt file in fsbrt-camera/neural_network/yolov7-ros/weights folder.
The folder located in neural_network/yolov7-ros/fsoco_training_results contains all training results and graphs.

ROS Node Setup & Usage

Before you launch the node, adjust the parameters in the launch file. For example, you need to set the path to your YOLOv7 weights, class labels text file path and the image topic to which this node should listen to. The launch file also contains a description for each parameter.

Launch the node:

roslaunch yolov7_ros yolov7.launch

Notes

• The detections are published using the BoundingBoxes message type.
• The detections will be published under /yolov7/detections.
• If you set the visualize parameter to true, the detections will be drawn into the image, which is then published under /yolov7/detections/visualization.

2D pixel to 3D point Deprojection

Using the FRAMOS Realsense D435e Camera, we can obtain the RGB Image and its respecting Depth Image which contains the depth value of each pixel. RGB Image gets sent to YOLOv7 NN and from it we obtain a list of all cones inside bounding boxes with its classes and 2D pixel coordinates.
Next, we use the rs2.rs2_deproject_pixel_to_point function from pyrealsense2 which returns x, y, and z coordinates for each cone.

Setup & Usage

Before you launch the node, adjust the parameters in the launch file. For example, you need to set the Depth Image and BoundingBoxes topics to subscribe to, as well as the Camera Info topic which contains intrinsic camera parameters. Next, set the topic to publish the PointCloud2 message that contains the color and 3D coordinates of the cones in meters.

Launch the node:

roslaunch 2D3D_deprojection deprojection.launch

The node synchronizes the Depth and BoundingBoxes topic using ROS Timestamps, ensuring the correct data extraction.

Frame Transformation

The origin of the coordinate system of resulting 3D coordinates is in the camera. For usage in the SLAM algorithm or sensor fusion, the points need to be transformed so the origin is in the vehicle's center of mass.

Camera position in reference to the CM (x is forward) :

x: -120mm
y: 0mm
z: 873.52mm

roll, pitch, yaw: 0 deg