Infant Pose Detection with Raspberry Pi's Streaming System
2023/12/01: 🥉과학기술정보통신부 주최2023 공개SW 개발자대회 동상(큐브리드 대표상) 수상2023/11/16: 🥈경기대학교2023 SW상상기업 프로그램 경진대회 우수상 수상2023/11/02 - 2023/11/04: ✌과학기술정보통신부 주최2023 SW인재 페스티벌 경기대학교 대표2023/08/16: 🏆경기대학교2023 SW상상기업 프로그램 경진대회 우수 사업계획서상 수상2023/06/14: 🥇경기대학교2023 AI컴퓨터공학 산학협력 프로젝트 경진대회 대상 수상
-
Predicts the baby's posture through Deep Learning - Pose Estimation technology and performs algorithm-based inappropriate posture detection based on the predicted posture.
-
When inappropriate posture is detected, a notification is sent to the user's app.
-
In this project, we used Raspberry Pi, camera module, v4l2rtspserver library as IoT means for video.
The code is developed using python 3.10 on Ubuntu 22.04. NVIDIA GPUs are needed. The code is developed and tested using two NVIDIA RTX 2080 SUPER GPU card. Other platforms or GPU cards are not fully tested.
- Clone this repository.
git clone https://github.com/LSTM2023/Infant-Pose-Detection- Set up a virtual environment such as venv or conda and install the requirements required.
python3 -m pip install -r requirements.txt- Make directory named firebase_cloud_messaging in '${Repo Root}/server_pose_estimation/firebase_cloud_messaging'.
mkdir firebase_cloud_messaging- Place the json file related to the FCM service account private key in the ${Repo Root}/server_pose_estimation/firebase_cloud_messaging for sending notifications .
${Repo Root}
├── raspberry_pi
└── server_pose_estimation
└── firebase_cloud_messaging
└── fcm_service_account_key.json- Create a file fcm.json, write it in the format shown below, and place it in the same directory, ${Repo Root}/server_pose_estimation/firebase_cloud_messaging.
{
"private_service_key": "fcm_service_account_key.json",
"emul_TOKEN": "TOKEN of Emulator",
"phone_TOKEN": "TOKEN of Phone Device",
"tablet_TOKEN": "TOKEN of Tablet Device"
}${Repo Root}
├── raspberry_pi
└── server_pose_estimation
└── firebase_cloud_messaging
├── fcm_service_account_key.json
└── fcm.json- Just run inference.py in the ${Repo Root}/server_pose_estimation directory.
python3 inference.py
- You can check how inference progresses!
- Please download SyRIP dataset from here. Download and extract them under ${Repo_ROOT}/server_pose_estimation/fine_tuning/dataset with the name of 'SyRIP_COCO', and make them look like this:
${Repo Root}
├── raspberry_pi
└── server_pose_estimation
└── fine_tuning
└── dataset
└── SyRIP_COCO
├── annotations
├── images
└── README.md- Convert the dataset from coco format annotations to yolo format labels using coco_to_yolo.py in ${Repo_ROOT}/server_pose_estimation/fine_tuning/dataset. You can specify the dataset to be converted by changing the path name specified in the code. Then, you can check the .txt format files containing the labels for each image in a directory named yolo_labels.
python3 coco_to_yolo.py- Create a SyRIP_YOLO directory in ${Repo_ROOT}/server_pose_estimation/fine_tuning/dataset and place the images and labels to be used for train and validation in it as follows.
${Repo Root}
├── firebase_cloud_messaging
├── raspberry_pi
└── server_pose_estimation
└── fine_tuning
└── dataset
├── SyRIP_COCO
└── SyRIP_YOLO
├── ${for_train}
│ ├── images
│ │ ├── train00001.jpg
│ │ ├── train00002.jpg
│ │ ├── ...
│ │ └── train10999.jpg
│ └── labels
│ ├── train00001.txt
│ ├── train00002.txt
│ ├── ...
│ └── train10999.txt
└── ${for_valid}
├── images
│ ├── test0.jpg
│ ├── test1.jpg
│ ├── ...
│ └── test499.jpg
└── labels
├── test0.txt
├── test1.txt
├── ...
└── test499.txt- After that, modify the path in SyRIP-pose.yaml to match ${for_train} and ${for_valid}.
train: ${for_train} # train images
val: ${for_valid} # val images- Just run train.py in the ${Repo Root}/server_pose_estimation/finetuning directory. You can edit You can edit You can edit You can edit You can edit You can edit
python3 train.py
- And you can check the performance of each model through valid.py.
python3 valid.py
- In our case, fine-tuning was performed by reconstructing the existing dataset as follows.
- The results of fine-tuning conducted with the reconstructed dataset are as follows.
We use YOLOv8-pose model from ultralytics to estimate the baby's pose, and use the coordinates of each body joint obtained through this to detect the baby's abnormal sleeping posture. The image below is an abnormal sleeping posture detection algorithm, and a detailed description of the algorithm is described in ${Repo Root}/server_pose_estimation/utils/pose_utils.py.
-
We used Debian 10, Buster version of Raspberry Pi 4B model, and used v4l2rtspserver library to transmit video to the server and app.
-
It is configured to measure temperature and humidity using the DHT11 sensor along with the camera module.
Infant-Pose-Detection is released under the AGPL-3.0 License.
@inproceedings{huang2021infant,
title={Invariant Representation Learning for Infant Pose Estimation with Small Data},
author={Huang, Xiaofei and Fu, Nihang and Liu, Shuangjun and Ostadabbas, Sarah},
booktitle={IEEE International Conference on Automatic Face and Gesture Recognition (FG), 2021},
month = {December},
year = {2021}
}@software{Jocher_YOLO_by_Ultralytics_2023,
author = {Jocher, Glenn and Chaurasia, Ayush and Qiu, Jing},
license = {AGPL-3.0},
month = jan,
title = {{YOLO by Ultralytics}},
url = {https://github.com/ultralytics/ultralytics},
version = {8.0.0},
year = {2023}
}