Detecting mouth with openCV

Question

I am trying to detect the mouth in an image with openCV, so I am using the following code:

#include "face_detection.h"

using namespace cv;

// Function detectAndDisplay
void detectAndDisplay(const std::string& file_name, cv::CascadeClassifier& face_cascade, cv::CascadeClassifier& mouth_cascade)
{
    Mat frame = imread(file_name);
    std::vector<Rect> faces;
    Mat frame_gray;
    Mat crop;
    Mat res;
    Mat gray;

    cvtColor(frame, frame_gray, COLOR_BGR2GRAY);
    equalizeHist(frame_gray, frame_gray);

    // Detect faces
    face_cascade.detectMultiScale(frame_gray, faces, 1.1, 3, 0 | CASCADE_SCALE_IMAGE, Size(30, 30));

    for(unsigned int i=0;i<faces.size();i++)
    {
        rectangle(frame,faces[i],Scalar(255,0,0),1,8,0);
        Mat face  = frame(faces[i]);
        cvtColor(face,face,CV_BGR2GRAY);
        std::vector <Rect> mouthi;
        mouth_cascade.detectMultiScale(face, mouthi);
        for(unsigned int k=0;k<mouthi.size();k++)
        {
        Point pt1(mouthi[k].x+faces[i].x , mouthi[k].y+faces[i].y);
        Point pt2(pt1.x+mouthi[k].width, pt1.y+mouthi[k].height);
        rectangle(frame, pt1,pt2,Scalar(0,255,0),1,8,0);
        }

    }

    imshow("Frame", frame);
    waitKey(33);
}

The classifiers are haarcascade_frontalface_alt.xml and haarcascade_mcs_mouth.xml.

The face is detected correctly but the mouth is not: I also obtain the eyes and some other parts, like the forehead. Is there a way to detect only the mouth?

minomic · Accepted Answer · 2015-04-18T21:06:13.550

6

I think I managed to solve the problem: focusing on the lower half of the face and increasing the scale factor did the trick and now I am able to detect the mouth with a good precision. Anyway this task seems much more complicated than face detection, even if I am using "simple" images, which means straight and full frontal.

Here are two examples: a success and a failure.

wrong

edited Apr 18 '15 at 21:06

answered Apr 18 '15 at 20:40

minomic

607
8
21

Hey, Can I get a link to the dataset from where you got these images? I have the images with me but I can't find the link to the page. I need to for citations. Thanks! – Anindit Karmakar May 06 '16 at 17:53
@AninditKarmakar Hi, it is the KDEF dataset: http://www.emotionlab.se/resources/kdef – minomic May 07 '16 at 05:08
@minomic Can you look on that topic: http://stackoverflow.com/questions/37921913/java-and-haarcascade-face-and-mouth-detection-mouth-as-the-nose – Adamo Jun 22 '16 at 09:08

score 0 · Answer 2 · edited Mar 25 '17 at 14:33

I was facing the same problem, so I focused only on the lower half of the face and created an ROI from the detected face. It looks something like this:

Mat ROI=image(Rect(face.x,face.y+face.height*0.6,face.width,face.height*0.3));

Where face is the detected face from the image.

This created an ROI from the detected face for the lower half only. Else the mouth detector was detecting the eyes also as mouth.

Then use the MouthCascade.xml from this link: http://alereimondo.no-ip.org/OpenCV/34 which is far more efficient than the inbuilt OpenCV one.

Detecting mouth with openCV

2 Answers2