The latest release of OpenCV (contrib) provide an interface to do face landmark detection. Although they have a brief document, it still took me a while to make it work. This tutorial works for those who are not familiar with the OpenCV but have successfully installed it once.
TL;DR: In terms of performance, I will recommend you to use dlib right now, which gives a much better performance for now.
1 > Compile OpenCV and OpenCV Contrib from source
Since we are going to user OpenCV Contrib, we have to compile the opencv ourselves rather than just install it from the package manager. The python version of OpenCV and OpenCV-contrib doesn’t support few features, one of which is the face landmark. Therefore, we are going for the C++ version of it. There are a lot of tutorials online to teach you how to install OpenCV with Contrib and I will just show you the steps briefly.
First, we need to download the source code. I used to go to their github page and click the `release` button for the latest release.
After downloading and unzipping, we have to install the dependencies that are used in OpenCV. I have to admit that I couldn’t find a complete dependency list without redundant packages. The most precise way is to check the CMake error and to install it one by one. To save your energy, I recommend part of this blog for reference.
With dependency properly installed, we can go the opencv source folder and create a `build/` directory. Inside it, I will suggest you to write a script to do the cmake command since I have seen like 20 versions of cmake command and always forgot which one I used last time. Keeping the configuration in a script can help you debug sometimes. My `cmake.sh` looks like:
cmake \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_CUDA=ON \
-D ENABLE_FAST_MATH=1 \
-D CUDA_FAST_MATH=1 \
-D WITH_CUBLAS=1 \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=/path/to/opencv_contrib-3.4.2/modules/ \
-D BUILD_SHARED_LIBS=ON \
-D WITH_GTK=ON \
-D BUILD_EXAMPLES=ON ..
Remember to change the OpenCV_contrib path to the correct one. Run `bash cmake.sh` to configure the compile environment.
Then `make all && sudo make install`.
Compiling can take quite a long time. It took about an hour on an i7-7700K (7 cores assigned).
2 > Configure and Compile
Sometimes you have to run `sudo ldconfig` to flush the installed library. To test whether the compiler can detect opencv, execute `pkg-config --cflags --libs opencv`. My result looks like:
-I/usr/local/include/opencv -I/usr/local/include -L/usr/local/lib -lopencv_stitching -lopencv_superres -lopencv_cudaobjdetect -lopencv_cudabgsegm -lopencv_videostab -lopencv_cudastereo -lopencv_cudacodec -lopencv_cudaoptflow -lopencv_cudalegacy -lopencv_cudafeatures2d -lopencv_cudawarping -lopencv_ccalib -lopencv_tracking -lopencv_xobjdetect -lopencv_dpm -lopencv_rgbd -lopencv_freetype -lopencv_hdf -lopencv_datasets -lopencv_saliency -lopencv_structured_light -lopencv_viz -lopencv_phase_unwrapping -lopencv_bioinspired -lopencv_aruco -lopencv_hfs -lopencv_text -lopencv_optflow -lopencv_fuzzy -lopencv_reg -lopencv_xphoto -lopencv_plot -lopencv_img_hash -lopencv_line_descriptor -lopencv_xfeatures2d -lopencv_shape -lopencv_ml -lopencv_bgsegm -lopencv_ximgproc -lopencv_dnn_objdetect -lopencv_dnn -lopencv_surface_matching -lopencv_face -lopencv_photo -lopencv_cudaimgproc -lopencv_cudafilters -lopencv_video -lopencv_objdetect -lopencv_cudaarithm -lopencv_stereo -lopencv_calib3d -lopencv_features2d -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_imgproc -lopencv_flann -lopencv_core -lopencv_cudev
Those flags are the ones we are going to use later.
The face landmark detection code locates at:
/path/to/opencv_contrib-3.4.2/modules/face/samples/sampleDetectLandmarksvideo.cpp
I recommend you to make a copy so that you can change its content without worrying about losing the original one.
The compile command is simple:
g++ ./sampleDetectLandmarksvideo.cpp `pkg-config --cflags --libs opencv` -o sampleDetectLandmarksvideo,
where `pkg-config --cflags --libs opencv` is flags that we tell the compiler to link with or to look headers from.
3 > Execution
We need the face_cascade and model_filename, which, sadly, no document tells you where to find them. Actually, you can find them in
`/path/to/opencv/data/lbpcascades/lbpcascade_frontalface.xml `
and
`/path/to/opencv/build/share/OpenCV/testdata/cv/face/face_landmark_model.dat`
Notice that we have to use the absolute path and the equality mark in `-c=..` cannot be omitted.
To be honest, I never successfully execute this file since I don’t have a video on the hard drive at that time. I modify the line 64 to `VideoCapture cap(0);` to make OpenCV read from my webcam and comment line 52 to 57 to disable the video path checking. Remember to compile it again.
After that my execution command looks like:
./sampleDetectLandmarksvideo -f=/path/to/installs/opencv/opencv-3.4.2/build/share/OpenCV/testdata/cv/face/face_landmark_model.dat -c=/path/to/opencv/opencv-3.4.2/data/lbpcascades/lbpcascade_frontalface.xml
As for the result, well, it’s pretty not robust and failed to detect my face with my glasses on. And it also shrinks your face in a weird way.
Followup: to remove the shrinking effect, comment line 86 in the source code and re-compile.
In terms of performance, I will recommend you to use dlib right now, which gives a much better performance for now.