This article is part of a series where I will be documenting my journey on the development of a social distancing feedback system for the blind as part of the OpenCV Spatial Competition. Check out the full series: Part 1, Part 2.

本文是系列文章的一部分，在該系列文章中，我將記錄我在OpenCV空間競賽中為盲人開發社交疏散反饋系統的過程。 查看完整的系列文章： 第1 部分 ， 第2部分 。

為什么要選擇面罩？ (Why Face Mask Detection?)

As mentioned in Part 1 of the series, the objective of this project is to develop a feedback system for the blind to help them maintain the social distance with the people around using the OAK-D (by the way, congratulations to Luxonis LLC and Satya Mallick for the successful Kickstarter campaign!). In this article, I will focus on the detection of the people around using Deep Learning models in order to keep the distance to those people.

如該系列的第1部分所述，該項目的目的是為盲人開發一個反饋系統，以幫助他們使用OAK-D保持與周圍人的社交距離 (順便恭喜， Luxonis LLC和Satya Mallick為成功的Kickstarter廣告系列！ )。 在本文中，我將重點介紹使用深度學習模型對周圍的人進行檢測，以使其與這些人保持距離。

The detection of the people around the user could be done in multiple ways, one option could be to train a model for pedestrian detection as in this repository. Another possible option could be to detect only faces instead of detecting the whole body. The benefit of face detection models is that due to the unique features of faces, it is easier to detect faces even without the need of Deep Learning models. For example, in this OpenCV tutorial a Haar feature-based cascade classifiers method is used to detect faces in real time even in low computation devices.

可以通過多種方式對用戶周圍的人進行檢測，一種選擇可以是像在該存儲庫中那樣為行人檢測訓練模型。 另一個可能的選擇是僅檢測面部而不是檢測整個身體。 人臉檢測模型的優勢在于，由于人臉的獨特功能，即使不需要深度學習模型，也更易于檢測人臉。 例如，在此OpenCV教程中 基于Haar特征的級聯分類器方法甚至可以在低計算設備中實時檢測人臉。

However, in this particular application, we also want to know whether the people around are wearing face mask or not. Then, the best option will be to use only one model that detect where the people are and if they are using face mask, i.e. a face mask detector.

但是，在此特定應用程序中，我們還想知道周圍的人是否戴著口罩。 然后，最好的選擇是僅使用一種模型來檢測人員所在的位置以及是否使用了面罩，即面罩檢測器。

面罩檢測器過多 (Overabundance of Face Mask Detectors)

Due to the increased use of face masks in the current pandemic, a high number of people have develop systems for face mask detection. A quick search in Github for “face mask detection” returns around 700 repositories on the topic. Similarly, a search for the same term in Youtube returns an endless list of videos showing the implementation of face detection models.

由于當前大流行中越來越多地使用面罩，因此許多人已經開發出用于面罩檢測的系統。 在Github中快速搜索“面罩檢測”將返回大約700個關于該主題的存儲庫。 同樣， 在Youtube中搜索同一詞會返回無盡的視頻列表，這些視頻顯示了人臉檢測模型的實現。

So, with such a high number of available examples, I expected that it should be quite easy to find an example that is fast enough for the OpenCV AI Kit with Depth (OAK-D) and that has good accuracy even in real life environments.

因此，由于有大量可用示例，我希望找到一個足夠快的帶深度的OpenCV AI套件 ( OAK-D )且即使在現實環境中也具有良好準確性的示例應該非常容易。

However, which face mask detection example should we choose?

但是， 我們應該選擇哪個面罩檢測示例？

First, I decided to look at the top starred face mask detection repositories in Github. In the table below, a list of datasets and models used in some of those repositories are summarized.

首先，我決定查看Github中最受關注的加星標口罩檢測存儲庫。 在下表中，匯總了其中一些存儲庫中使用的數據集和模型的列表。

As it can be observed, there are two main ways to perform face mask detection: 1. Face detection + face mask classification on each detected face or 2. Directly perform face mask detection. The first method could have better accuracy because already available face detection models have been trained in thousands of face images. In comparison, as ashown in the table, the face mask detection datasets have fewer images for training, where the AIZOOTech dataset has the higher number of images.

可以看出，有兩種主要的方法來執行面罩檢測： 1.面部檢測+在每個檢測到的面部上進行面罩分類或2.直接執行面罩檢測 。 第一種方法可能具有更好的準確性，因為已經在數千張面部圖像中訓練了已有的面部檢測模型。 相比之下，如表中所示，面罩檢測數據集具有較少的訓練圖像，其中AIZOOTech數據集具有較高的圖像數目。

However, most of previous face detection models are trained on faces that were not covered in most of the cases. For this reason, in cases where the face is covered with a mask, the face detection model could miss the detection of the faces (as explained in the very well detailed article in pyimagesearch.com).

但是，大多數以前的面部檢測模型都是針對大多數情況下未涵蓋的面部進行訓練的。 因此，在人臉被面具覆蓋的情況下，人臉檢測模型可能會錯過人臉的檢測(如pyimagesearch.com中非常詳細的文章所述 )。

使用OAK-D進行口罩檢測 (Face mask detection using OAK-D)

Before analyzing the previously mentioned examples, the people at Luxonis already provide a demo for face mask detection using the OAK-D. The model in the demo is a MobileNetV2 (SSD) trained in Google Colab.

在分析前面提到的示例之前， Luxonis的人員已經提供了使用OAK-D進行面罩檢測的演示 。 該演示中的模型是經過Google Colab培訓的MobileNetV2(SSD)。

Note: Even though it is not included in the demo, they also provide another Google Colab script for training a YOLOv3-tiny model for face mask detection.

注意：盡管演示中未包含該工具，但他們還提供了另一個Google Colab腳本，用于訓練YOLOv3-tiny模型以檢測面罩 。

To run the demo, it is necessary to install the DepthAI Python module. As mentioned in Part 1, the Windows version is still experimental. However, recently the process has been updated so that it is much easier to install the library following the steps in the last comment in the Luxonis Dicussion.

要運行該演示，必須安裝DepthAI Python模塊。 如第1部分所述，Windows版本仍處于試驗階段。 但是，最近該過程已更新，因此按照Luxonis Dicussion中 最后一個注釋中 的 步驟安裝該庫要容易得多。

However, as of today, the face mask detection demo is configured to work with an older version of the DepthAI library. So, I modified the demo to work with the current version of DepthAI and can be found in my Github repository for this project. To run the demo, it is necessary to install the DepthAI library as explained before and add the depthai folder to the PYTHONPATH of your system. Next open the command and run the following commands:

但是，截至今天，面罩檢測演示已配置為可與DepthAI庫的舊版本一起使用。 因此，我修改了演示以與DepthAI的當前版本一起使用 ，可以在我的 這個項目的Github倉庫 。 要運行該演示，必須按照前面的說明安裝DepthAI庫，并將depthai文件夾添加到系統的PYTHONPATH中。 接下來打開命令并運行以下命令：

git clone https://github.com/ibaiGorordo/Social-Distance-Feedback.git
cd "Social-Distance-Feedback\Part 2 - Mask Detection"
python demo_mask_detector.py

The demo_mask_detector.py is a script that configures the OAK-D to perform face mask detection on RGB camera and shows the image and detection from the OAK-D.

demo_mask_detector.py是一個腳本，該腳本將OAK-D配置為在RGB相機上執行面部遮罩檢測，并顯示圖像和來自OAK-D的檢測。

import numpy as np
import cv2
from demo_helpers import config, capture_image, get_detection, calculate_frame_speed, decode_mobilenet_ssd, show_mobilenet_ssd
from time import time, sleep, monotonic
import os
import depthai
print('Using depthai module from: ', depthai.__file__)# Create a list of enabled streams ()
stream_names = ['metaout', 'previewout']device = depthai.Device('', False)# create the pipeline, here is the first connection with the device
p = device.create_pipeline(config=config)if p is None:print('Pipeline is not created.')exit(3)while True:# retreive data from the device# data is stored in packets, there are nnet (Neural NETwork) packets which have additional functions for NNet result interpretationnnet_packets, data_packets = p.get_available_nnet_and_data_packets(True)ret, frame = capture_image(data_packets)nnet_prev = get_detection(nnet_packets)if ret:frame_count = calculate_frame_speed()nn_frame = show_mobilenet_ssd(nnet_prev["entries_prev"]['rgb'], frame, is_depth=0)cv2.putText(nn_frame, "fps: " + str(frame_count), (25, 50), cv2.FONT_HERSHEY_SIMPLEX, 1.0, (0, 0, 0))cv2.imshow("Mask detection", nn_frame)key = cv2.waitKey(1)if key == ord('q'):breakdel p  # in order to stop the pipeline object should be deleted, otherwise device will continue working. This is required if you are going to add code after the main loop, otherwise you can ommit it.
device.deinit_device()
print('py: DONE.')

The video below shows the inference output from the OAK-D for the MobileNetV2 (SSD) face mask detection model (trained using DepthAI’s Google Colab script).

以下視頻顯示了OAK-D對MobileNetV2(SSD) 面罩檢測模型的推理輸出(使用DepthAI的Google Colab密碼進行了訓練)。

面罩檢測模型在野外如何表現？ (How do face mask detection models behave in the wild?)

The previous example is similar to the many tutorials that end up with an example of the inference using the webcam. However, the objective of my system is to be used in daily life, particularly outdoors. Therefore, the system should be robust in different light conditions and even when there are multiple people around.

前面的示例與許多教程相似，最后以使用網絡攝像頭進行推理為例。 但是，我系統的目標是在日常生活中，尤其是在戶外使用。 因此，即使在周圍有多個人的情況下，該系統也應在不同的光照條件下都非常堅固。

For that reason, I decided to see how different face mask detection models behaved in a more realistic environment. For that purpose, I used this public video from pexels.com of people walking in a night market. The video below shows a comparison of the face mask detection for the SSD-MobileNetV2 Vs. the YOLOv3-tiny models trained using DepthAI’s Google Colab scripts. The code for the inference program can be found in my Github repository here.

因此，我決定看看在更現實的環境中不同的面罩檢測模型如何表現。 為此，我使用了來自pexels.com的公開視頻，內容是在夜市里散步的人們 。 以下視頻比較了SSD-MobileNetV2 Vs的面罩檢測。 使用DepthAI的Google Colab腳本訓練的YOLOv3-tiny模型。 推理程序的代碼可以在我的Github存儲庫中找到 。

As it can be observed, the SSD-MobilenetV2 model had a higher number of detections but as a result, a higher number of those detections were wrong detections. Even increasing the confidence threshold to 0.7 (as in the video above), the SSD-MobilenetV2 model still had a high number of false detections.

可以觀察到， SSD-MobilenetV2模型具有較高的檢測數量，但結果是，這些檢測中的較高數量是錯誤檢測。 即使將置信度閾值提高到0.7(如上面的視頻中所示)， SSD-MobilenetV2模型仍然存在大量錯誤檢測。

On the other hand, the YOLOv3-tiny model missed some of the faces (particularly the faces that were far away) but had a more stable detections with a confidence threshold of 0.5. Since our application only needs to detect people that are close to the user (at a distance of 3 meters or closer), the YOLOv3-tiny model seems to be the most promising model of the two.

另一方面， YOLOv3-tiny模型遺漏了一些面部(尤其是距離較遠的面部)，但是檢測更為穩定，置信度閾值為0.5。 由于我們的應用程序僅需要檢測靠近用戶(3米或更近的距離)的人員，因此YOLOv3-tiny模型似乎是兩者中最有希望的模型。

YOLOv4呢？ (What about YOLOv4?)

Recently, Alexey Bochkovskiy presented a new YOLO version (YOLOv4) which provides higher performance than the previous versions. In the video below, there is an example from Mladen Zamanov where it can be seen that YOLOv4 can perform face mask detection even with many people around.

最近，Alexey Bochkovskiy提出了 YOLO新版本(YOLOv4) 比以前的版本提供更高的性能。 在下面的視頻中，有一個來自Mladen Zamanov的示例，可以看出YOLOv4即使在周圍很多人的情況下也可以執行面部遮罩檢測。

The problem of using YOLOv4 in our application is that in order to pass the model to the OAK-D, the model needs to be converted to a .blob file to be run by the Myriad X inside the OAK-D. However, in order to convert the model, it is necessary to use OpenVINO toolkit’s model optimizer which currently officially supports until YOLOv3.

在我們的應用程序中使用YOLOv4的問題是，為了將模型傳遞給OAK-D ，需要將模型轉換為.blob文件 ，以由OAK-D內部的Myriad X運行。 但是，為了轉換模型，必須使用OpenVINO工具箱的模型優化器 ，該模型優化器目前在YOLOv3之前一直受官方支持。

Even though that thanks to the repository by TNTWEN it is possible to use the model optimizer with the latest OpenVINO (2020.4) version, the depthAI module still does not support it. As a result, as of today, it seems that YOLOv4 cannot be used in the OAK-D with the built-in cameras.

即使由于TNTWEN的存儲庫，仍可以將模型優化器與最新的OpenVINO(2020.4)版本一起使用 ， depthAI模塊仍不支持它。 結果，從今天開始，似乎YOLOv4無法在內置相機的OAK-D中使用。

Anyway, I decided to give a try to YOLOv4 in case in a near future it becomes available for the OAK-D. Particularly, I focused on the new YOLOv4-tiny version because it should be more suitable for real time inference in devices with lower computation power (actually the author of YOLOv4 has been able to run a modified version of the full YOLOv4 in the same chip inside the OAK-D as shown here).

無論如何，我決定嘗試一下YOLOv4 ，以防它在不久的將來可用于OAK-D 。 特別是，我專注于新版YOLOv4-tiny版本，因為它應該更適合于具有較低計算能力的設備中的實時推斷(實際上YOLOv4的作者已經能夠在同一芯片內部運行完整版YOLOv4的修改版) OAK-D ，如此處所示 )。

For that purpose, I trained the YOLOv4-tiny model in Google Colab using this script (code below), which was based on DepthAI’s original script. In the same script, I also added the code to detect face masks on the same video I used to compare SSD-MobileNetV2 with YOLOv3.

為此，我使用了該腳本 (以下代碼) 在Google Colab中訓練了YOLOv4-tiny模型， 該腳本基于DepthAI的原始腳本。 在同一腳本中，我還添加了用于檢測同一視頻的面罩的代碼，該視頻用于比較SSD-MobileNetV2和YOLOv3 。

In the video below, I show the comparison of the results for YOLOv3-tiny Vs. YOLOv4-tiny for face mask detection.

在下面的視頻中，我顯示了YOLOv3-tiny Vs結果的比較。 YOLOv4-tiny用于面罩檢測 。

As it can be observed, the results very similar between both models. In some cases, YOLOv4-tiny is capable of detecting faces that YOLOv3-tiny is not capable, but overall the results are almost identical.

可以看出，兩個模型之間的結果非常相似。 在某些情況下， YOLOv4-tiny能夠檢測到YOLOv3-tiny無法檢測到的面部，但是總體結果幾乎是相同的。

In summary, I will probably continue using YOLOv3-tiny for face mask detection unless new support becomes available for YOLOv4. In the next part, I will focus on the combination of depth and object detection data.

總之，除非可以為YOLOv4提供新的支持，否則我可能會繼續使用YOLOv3-tiny進行面罩檢測。 在下一部分中，我將重點介紹深度和對象檢測數據的組合。

All the code from this part and for the next part can be found in my following repository: https://github.com/ibaiGorordo/Social-Distance-Feedback.

該部分以及下一部分的所有代碼都可以在我的以下存儲庫中找到： https : //github.com/ibaiGorordo/Social-Distance-Feedback 。