-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathindex.html
More file actions
215 lines (197 loc) · 10.4 KB
/
Copy pathindex.html
File metadata and controls
215 lines (197 loc) · 10.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>PyAFAR: Python-based Automated Facial Action Recognition library for use in Infants and Adults</title>
<!-- Bootstrap -->
<link href="./css/bootstrap-4.4.1.css" rel="stylesheet">
<link href="./css/project.css" rel="stylesheet">
<link rel="stylesheet" href="./css/font-awesome.min.css">
</head>
<!-- cover -->
<body>
<section>
<div class="jumbotron text-center mt-0">
<div class="section logos" style="text-align:center">
<IMG src="./images/Pitt.png" height="60" border="0">
</td>        
<IMG src="./images/universiteit-utrecht-logo.png" height="75"
border="0"></td>        
<IMG src="./images/University-Of-Miami-Symbol.png" height="60" border="0"></td>        
<!-- <IMG src="https://acii-conf.net/2023/wp-content/uploads/2022/11/cropped-cropped-acii2023-icon-k.png" height="60" border="0"> -->
</td>
</div></p>
<div class="col-12">
<h2>PyAFAR: Python-based Automated Facial Action Recognition library for use in Infants and Adults</h2>
<h5><strong>(FG 2024 & ACII 2023)</strong></h5>
<p></p>
<h6>
<div class="authors">
<a href="https://saurabhh.com/" target="_blank">Saurabh Hinduja</a><sup> 1</sup><sup> 4</sup>  
<a href="https://itironal.github.io/" target="_blank">Itir Onal Ertugrul</a><sup> 2</sup>  
<a href="https://bmaneesh.github.io/" target="_blank">Maneesh Bilalpur </a><sup> 1</sup>  
<a href="https://scholar.google.com/citations?user=CFpUEVwAAAAJ&hl=en" target="_blank"> Daniel S. Messinger</a><sup> 3</sup>  
<a href="https://www.jeffcohn.net/" target="_blank">Jeffrey F. Cohn</a><sup> 1</sup>  
</div>
<div class="affiliations">
<sup>1</sup>University of Pittsburgh  
<sup>2</sup>Utrecht University  
<sup>3</sup>University of Miami  
<sup>4</sup>CGI Technologies and Solutions Inc  
</div>
<br>
<div class="row justify-content-center">
<div class="column">
<p class="mb-5"><a class="btn btn-large btn-light" href="https://www.jeffcohn.net/wp-content/uploads/2023/08/ACII_2023_paper_242-2.pdf" role="button" target="_blank">
<i class="fa fa-file"></i> Paper</a> </p>
</div>
<div class="column">
<p class="mb-5"><a class="btn btn-large btn-light" href="https://github.com/AffectAnalysisGroup/PyAFAR/wiki/1.-Home" role="button" target="_blank">
<i class="fa fa-book"></i> Docs</a> </p>
</div>
<div class="column">
<p class="mb-5"><a class="btn btn-large btn-light" href="https://github.com/AffectAnalysisGroup/PyAFAR/wiki/3.-Installation" role="button" target="_blank">
<i class="fa fa-download"></i> Install</a> </p>
</div>
</div>
</div>
</div>
</section>
<section>
<div class="col-12 text-center">
<h5><font color="red">PyAFAR has disabled person tracking as MediaPipe dependency no longer supports python 3.7. It will be fixed in a future release.</font></h5>
</div>
</section>
<section>
<div class="jumbotron-grey">
<div class="row">
<div class="col-12 text-center">
<h3><b>Abstract</b></h3>
<br>
<div class="row" style="margin-bottom:5px">
</div>
<p class="text-left">
PyAFAR is a Python-based, open-source facial action unit detection library for use with adults and infants. Convolutional Neural Networks were trained on BP4D+ for adults, and Histogram of Gradients (HoG) features from MIAMI and CLOCK databases for infants were used with Light Gradient Boosting Machines. In adults, Action Unit occurrence and intensity detection are enabled for 12 AUs. The 12 AUs were selected on the criterion that they occurred more than 5% of the time in the training data that included BP4D. Because 5% baseline was the minimum for which reliability could be measured with confidence, AU with prevalence lower than that were not included. Action unit intensity estimation is enabled for 5 of these AU. In infants, AU occurrence is enabled for 9 action units that are involved in expression of positive and negative affect. For both adults and infants, facial landmark and head pose tracking are enabled as well. For adults, multiple persons within a video may be tracked. The library is developed for ease of use. The models are available for fine-tuning and further training. PyAFAR may be easily incorporated into user Python code.</p>
<div class="row" style="margin-bottom:5px">
<div class="col" style="text-align:center">
<img class="thumbnail" src="./images/pyafar_pipeline_updated.jpg" style="width:70%; margin-bottom:0px">
</div>
</div>
<p class="text-fig">Pipeline of PyAFAR.
</p>
<p>PyAFAR has separate modules for facial feature extraction and tracking, Face normalization and AU predictions:</p>
<p class="text-left">It uses <a href="https://research.google/pubs/pub48292/">MediaPipe</a> library for landmarks. <s>Tracking is performed using the <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Schroff_FaceNet_A_Unified_2015_CVPR_paper.pdf">FaceNet</a></s>. The Perspective-n-Point (PnP) method is used to predict Roll, Pitch and Yaw.</p>
<p class="text-left"> The landmark predictions are used to normalize faces using the <a href="http://dlib.net/">dlib</a> library.</p>
<p class="text-left"> Normalized faces are used for AU predictions (occurrence and intensity). Separate detection modules for occurrence are available for adults and infants. Intensity predictions are available for adults only.</p>
<p class="text-left"> PyAFAR can output frame-level predictions in CSV and JSON formats to enable easy reading with most platforms used by both computational as well as domain experts.</p>
</p>
</div>
</div>
</div>
</section>
<br>
<section>
<div class="container">
<div class="row">
<div class="col-12 text-center">
<h3><b>Demo</b></h3>
<!-- <iframe width="560" height="315" src="./images/pyafar_demo.git" title="YouTube video player" frameborder="0" allowfullscreen></iframe> -->
<!-- added demo video -->
<div class="col" style="text-align:center">
<img class="thumbnail" src="./images/pyafar_demo.gif" style="width:80%; margin-bottom:20px">
<p>For installation instructions click <a href="https://github.com/AffectAnalysisGroup/PyAFAR/wiki/3.-Installation">here</a> and how to use click <a href="https://github.com/AffectAnalysisGroup/PyAFAR/wiki/4.-How-to-use">here</a>.</p>
</div>
<p>Interested in learning more about Facial Action Coding System (FACS) for Action Unit annotation? Visit this <a href="https://sites.pitt.edu/~jeffcohn/FACSmodule.html">interactive application</a></p>
</div>
</div>
</div>
</section>
<br>
<!-- <section>
<div class="container">
<div class="row">
<div class="col-12 text-center">
<h3><b>Installation</b></h3>
<p>CPU installation </p>
<code><pre style="background-color: #e9eeef">
pip install https://github.com/AffectAnalysisGroup/PyAFAR/releases/download/v2.0.0/PyAFAR_CPU-2.0-py3-none-any.whl
</code></pre>
<p>GPU installation </p>
<code><pre style="background-color: #e9eeef">
pip install https://github.com/AffectAnalysisGroup/PyAFAR/releases/download/v2.0.0/PyAFAR_GPU-2.0-py3-none-any.whl
</code></pre>
</div>
</div>
</div>
</div>
</section>
<br> -->
<section>
<div class="container">
<div class="row">
<div class="col-12 text-center">
<h3><b>How to use</b></h3>
<div class="col" style="text-align:center">
<img class="thumbnail" src="./images/how_to_use.gif" style="width:80%; margin-bottom:20px">
</div>
</div>
</div>
</div>
</section>
<br>
<br>
<section>
<div class="container">
<div class="row ">
<div class="col-12">
<h3><b>Citations</b></h3>
<pre style="background-color: #e9eeef">
<code>
@inproceedings{hindujapyafar,
title={PyAFAR: Python-based Automated Facial Action Recognition library for use in Infants and Adults},
author={Hinduja, Saurabh and Ertugrul, Itir Onal and Bilalpur, Maneesh and Messinger, Daniel S and Cohn, Jeffrey F},
booktitle={International Conference on Affective Computing and Intelligent Interaction},
year={2023},
organization={IEEE}
}
@inproceedings{ertugrulpyafar,
title={Expanding PyAFAR: A Novel Privacy-Preserving Infant AU Detector},
author={Ertugrul, Itir Onal and Hinduja, Saurabh and Bilalpur, Maneesh and Messinger, Daniel S and Cohn, Jeffrey F},
booktitle={International Conference on Automatic Face and Gesture Recognition},
year={2024},
organization={IEEE}
}
</code></pre>
</div>
</div>
</div>
</section>
<section>
<div class="container">
<div class="row">
<div class="col-12">
<h3><b>Acknowledgements</h3></b></h3>
<p class="text-left">The development of PyAFAR was supported in part by NIH awards R01MH096951, R01GM105004, and UH3NS100549.</p>
<br>
</div>
</div>
</div>
</div>
</section>
<section>
<div class="container">
<div class="row ">
<div class="col-12">
<h3><b>License</b></h3>
PyAFAR is freely available for free non-commercial use, and may be redistributed under these conditions. Please see the
<a href="https://github.com/AffectAnalysisGroup/PyAFAR?tab=readme-ov-file#license"> complete licensing information</a>. Interested in a commercial license? Please contact <a href="https://www.jeffcohn.net/">Jeffrey Cohn</a>.
</div>
</div>
</div>
</section>
</div>
</div>
</body>
</html>