rotated face detection

Question

Is there a library for detecting faces that have been rotated in the image plane? Or is there some way in which I could use a cascade for upright face detection with opencv to do it?

Answer 1

Here's a simple one I wrote with Python cv2

It's not the most efficient thing, and it uses the naive way suggested by etarion, but it works fairly well for just normal head tilting (It detect anything from -40 to 40 head tilt, which is roughly as much as you can tilt your head staying upright.

import cv2
from math import sin, cos, radians

camera =  cv2.VideoCapture(0)
face = cv2.CascadeClassifier("haarcascade_frontalface_alt2.xml")

settings = {
    'scaleFactor': 1.3, 
    'minNeighbors': 3, 
    'minSize': (50, 50), 
    'flags': cv2.cv.CV_HAAR_FIND_BIGGEST_OBJECT|cv2.cv.CV_HAAR_DO_ROUGH_SEARCH
}

def rotate_image(image, angle):
    if angle == 0: return image
    height, width = image.shape[:2]
    rot_mat = cv2.getRotationMatrix2D((width/2, height/2), angle, 0.9)
    result = cv2.warpAffine(image, rot_mat, (width, height), flags=cv2.INTER_LINEAR)
    return result

def rotate_point(pos, img, angle):
    if angle == 0: return pos
    x = pos[0] - img.shape[1]*0.4
    y = pos[1] - img.shape[0]*0.4
    newx = x*cos(radians(angle)) + y*sin(radians(angle)) + img.shape[1]*0.4
    newy = -x*sin(radians(angle)) + y*cos(radians(angle)) + img.shape[0]*0.4
    return int(newx), int(newy), pos[2], pos[3]

while True:
    ret, img = camera.read()

    for angle in [0, -25, 25]:
        rimg = rotate_image(img, angle)
        detected = face.detectMultiScale(rimg, **settings)
        if len(detected):
            detected = [rotate_point(detected[-1], img, -angle)]
            break

    # Make a copy as we don't want to draw on the original image:
    for x, y, w, h in detected[-1:]:
        cv2.rectangle(img, (x, y), (x+w, y+h), (255,0,0), 2)

    cv2.imshow('facedetect', img)

    if cv2.waitKey(5) != -1:
        break

cv2.destroyWindow("facedetect")

Answer 2

Personally, I don't know of a library. But, what I can say is, use an eye detection Haar Cascade, and draw a line between the eyes. Then, you can use the atan function and find the angle by which the head is rotated. (Assuming that the person has both eyes on the same horizontal level when head is not rotated)

deg = atan( (leftEye.y - rightEye.y) / (leftEye.x - rightEye.x) )

Once you get this angle, rotate the image you have by negative deg degrees and you should have a face which can be detected using the Haar Cascades.

Answer 3

Naive way:

• Generate list of angles (for example, from -170 to 180 in 10 degree steps)
• For each angle n in the list:
  • Rotate image by n degrees
  • Run face detector on rotated image
  • Compute the position of the detected faces in the original image (undo the rotation)
• Perform non-maximum suppression on the joined result from all angles (you will likely get multiple detections from neighbouring angles)

Answer 4

you can use bag of words/bag of features method with constrains AAM,ASM methods. but they also can give not optimal solution converges not to global maximum.

also haar-like-features are just collection of features and you can use rotation invariant features and put it then in adaboost classifer.

Answer 5

I had been dealing with the same problem of face detection for non-frontal images. Try using Multi Task CNN. It's the best solution for face detection and alignment. It is able to deal with problems like various poses, lighting, occlusion.

The paper is available at Link. The code is available on GitHub at Link. I used the python implementation and the results are outstanding. Although the code is a little slow if the image has a lot of faces.

Although if you want to stick to OpenCV, then a new deep learning model for face detection has been added to OpenCV. The results are not as good as Multi Task CNN. There's an implementation of OpenCV Deep Learning Model for Face Detection at pyimagesearch Link

Answer 6

mtcnn works great. It seems it has only issue when face is very near to 90 degree or 180 degree. SO if normal detection fails, just rotate the image by 45 degrees, and try again. If there is a face in the image, then this should detect it.

I am curious though, why does mtcnn fails when face is exactly 90 degree rotated or inverted (180 degrees rotated)

Answer 7

This repo can detect objects as rotated bounding boxes: https://github.com/NVIDIA/retinanet-examples

You could create a dataset from Open Images by randomly rotating images containing the 'human faces' class by -30 to 30 degrees, then train this network to detect those faces.

Answer 8

Methods for face detection based on color histogram are independent of face orientation.