opencv - 如何在“处理”中使用网络摄像头确定上唇和下唇之间的距离？

标签 opencv image-processing processing video-processing face-detection

我应该从哪里开始？我可以看到使用Python，Java脚本进行的大量人脸识别和分析，但是处理效果如何？

我想通过网络摄像头在上下唇之间的最高点和最低点之间使用2个点来确定距离，以在以后的项目中使用它。

任何帮助，将不胜感激

最佳答案

如果要单独在“处理”中执行此操作，则可以使用Greg Borenstein's OpenCV for Processing library:

  • 您可以以Face Detection example
    • 开头
  • 一旦检测到脸部，就可以使用OpenCV.CASCADE_MOUTH在脸部矩形内检测到嘴巴。
  • 一旦检测到嘴巴，也许可以使用嘴巴边界框的高度。有关更多详细信息，请使用OpenCV对该矩形设置阈值。希望张开的嘴巴能与皮肤的其他部分很好地分开。 Finding contours应该为您提供可以使用的要点列表。

    • 要获得更精确的信息，可以使用Jason Saragih的CLM FaceTracker, which is available as an OpenFrameworks addon。 OpenFrameworks与处理相似。如果在处理中确实需要这种准确性，则可以在后台运行FaceOSC并在使用oscP5的处理中读取口坐标

    更新

    对于第一种选择，使用HAAR级联分类器，结果有两个问题:
  • OpenCV处理库可以加载一个级联，第二个实例将覆盖第一个实例。
  • OpenCV.CASCADE_MOUTH似乎对于闭口效果更好，但对张开嘴效果不佳

    • 要解决第一个问题，您可以直接使用OpenCV Java API，绕过OpenCV处理进行多个级联检测。

    有几个参数可以帮助检测，例如在手传到分类器之前先了解一下嘴的边界框。
    我已经使用便携式计算机上的网络摄像头完成了一项基本测试，并测量了各种距离下脸部和嘴部的边界框。这是一个例子:
    import gab.opencv.*;
import org.opencv.core.*;
import org.opencv.objdetect.*;

import processing.video.*;

Capture video;
OpenCV opencv;

CascadeClassifier faceDetector,mouthDetector;
MatOfRect faceDetections,mouthDetections;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
Size minFeatureSizeFace = new Size(50,60);
Size maxFeatureSizeFace = new Size(125,150);
Size minFeatureSizeMouth = new Size(30,10);
Size maxFeatureSizeMouth = new Size(120,60);

// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 4;
//laptop webcam face rectangle
//far, small scale, ~50,60px
//typing distance, ~83,91px
//really close, ~125,150
//laptop webcam mouth rectangle
//far, small scale, ~30,10
//typing distance, ~50,25px
//really close, ~120,60

int mouthHeightHistory = 30;
int[] mouthHeights = new int[mouthHeightHistory]; 

void setup() {
  opencv = new OpenCV(this,320,240);
  size(opencv.width, opencv.height);
  noFill();
  frameRate(30);

  video = new Capture(this,width,height);
  video.start();

  faceDetector = new CascadeClassifier(dataPath("haarcascade_frontalface_alt2.xml"));
  mouthDetector = new CascadeClassifier(dataPath("haarcascade_mcs_mouth.xml"));

}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);
  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  //detect face using raw Java OpenCV API
  Mat equalizedImg = opencv.getGray();
  faceDetections = new MatOfRect();
  faceDetector.detectMultiScale(equalizedImg, faceDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeFace, maxFeatureSizeFace);
  Rect[] faceDetectionResults = faceDetections.toArray();
  int faces = faceDetectionResults.length;
  text("detected faces: "+faces,5,15);
  if(faces >= 1){
    Rect face = faceDetectionResults[0];
    stroke(0,192,0);
    rect(face.x,face.y,face.width,face.height);
    //detect mouth - only within face rectangle, not the whole frame
    Rect faceLower = face.clone();
    faceLower.height = (int) (face.height * 0.65);
    faceLower.y = face.y + faceLower.height; 
    Mat faceROI = equalizedImg.submat(faceLower);
    //debug view of ROI
    PImage faceImg = createImage(faceLower.width,faceLower.height,RGB);
    opencv.toPImage(faceROI,faceImg);
    image(faceImg,width-faceImg.width,0);

    mouthDetections = new MatOfRect();
    mouthDetector.detectMultiScale(faceROI, mouthDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeMouth, maxFeatureSizeMouth);
    Rect[] mouthDetectionResults = mouthDetections.toArray();
    int mouths = mouthDetectionResults.length;
    text("detected mouths: "+mouths,5,25);
    if(mouths >= 1){
      Rect mouth = mouthDetectionResults[0];
      stroke(192,0,0);
      rect(faceLower.x + mouth.x,faceLower.y + mouth.y,mouth.width,mouth.height);
      text("mouth height:"+mouth.height+"~px",5,35);
      updateAndPlotMouthHistory(mouth.height);
    }
  }
}
void updateAndPlotMouthHistory(int newHeight){
  //shift older values by 1
  for(int i = mouthHeightHistory-1; i > 0; i--){
    mouthHeights[i] = mouthHeights[i-1];
  } 
  //add new value at the front
  mouthHeights[0] = newHeight;
  //plot
  float graphWidth = 100.0;
  float elementWidth = graphWidth / mouthHeightHistory;  
  for(int i = 0; i < mouthHeightHistory; i++){
    rect(elementWidth * i,45,elementWidth,mouthHeights[i]);
  }
}
void captureEvent(Capture c) {
  c.read();
}

    需要做的非常难忘的注释:我已将级联xml文件从OpenCV处理库文件夹(~/Documents/Processing/libraries/opencv_processing/library/cascade-files)复制到草图的数据文件夹中。我的草图是OpenCVMouthOpen，因此文件夹结构如下所示:
    OpenCVMouthOpen
├── OpenCVMouthOpen.pde
└── data
    ├── haarcascade_frontalface_alt.xml
    ├── haarcascade_frontalface_alt2.xml
    ├── haarcascade_frontalface_alt_tree.xml
    ├── haarcascade_frontalface_default.xml
    ├── haarcascade_mcs_mouth.xml
    └── lbpcascade_frontalface.xml

    如果您不复制级联文件并按原样使用代码，则不会出现任何错误，但是检测根本无法进行。如果要检查，可以做
    println(faceDetector.empty())

    在setup()函数的末尾，如果您获得false，则级联已加载，如果您获得true，则级联尚未加载。

    您可能需要针对面部和嘴巴使用minFeatureSize和maxFeatureSize值进行设置。第二个问题，级联不能很好地检测到张开嘴是棘手的。可能已经有训练有素的级联来张开嘴巴，但是您需要找到它。否则，使用这种方法，您可能需要自己进行训练，这可能会有些乏味。

    但是，请注意，当检测到嘴巴时，左侧会绘制一个上下颠倒的图。在我的测试中，我注意到高度不是非常准确，但是图形中有明显的变化。您可能无法获得稳定的嘴巴高度，但是通过将当前高度与以前的平均高度值进行比较，您应该会看到一些峰值(该值从正到负，反之亦然)，这使您对嘴巴张开/合拢有所了解。

    尽管在整个图像中搜索一张嘴巴而不是一张脸可能会比较慢且准确性不高，但这是一个更简单的设置。如果您对项目的准确性降低，误报率更高，则可能会更简单:
    import gab.opencv.*;
import java.awt.Rectangle;
import org.opencv.objdetect.Objdetect;
import processing.video.*;

Capture video;
OpenCV opencv;
Rectangle[] faces,mouths;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
int minFeatureSize = 20;
int maxFeatureSize = 150;
// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 6;

void setup() {
  size(320, 240);
  noFill();
  stroke(0, 192, 0);
  strokeWeight(3);

  video = new Capture(this,width,height);
  video.start();

  opencv  = new OpenCV(this,320,240);
  opencv.loadCascade(OpenCV.CASCADE_MOUTH);
}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);
  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  Rectangle[] mouths = opencv.detect(searchScaleFactor,minNeighbors,flags,minFeatureSize, maxFeatureSize);
  for (int i = 0; i < mouths.length; i++) {
    text(mouths[i].x + "," + mouths[i].y + "," + mouths[i].width + "," + mouths[i].height,mouths[i].x, mouths[i].y);
    rect(mouths[i].x, mouths[i].y, mouths[i].width, mouths[i].height);
  }
}
void captureEvent(Capture c) {
  c.read();
}

    我也提到了分段/阈值。这是一个粗略的示例，使用检测到的脸部的下部仅是一个基本阈值，然后使用一些基本形态过滤器(腐 eclipse /扩张)来稍微清理阈值图像:
    import gab.opencv.*;
import org.opencv.core.*;
import org.opencv.objdetect.*;
import org.opencv.imgproc.Imgproc;
import java.awt.Rectangle;
import java.util.*;

import processing.video.*;

Capture video;
OpenCV opencv;

CascadeClassifier faceDetector,mouthDetector;
MatOfRect faceDetections,mouthDetections;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
Size minFeatureSizeFace = new Size(50,60);
Size maxFeatureSizeFace = new Size(125,150);

// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 4;
//laptop webcam face rectangle
//far, small scale, ~50,60px
//typing distance, ~83,91px
//really close, ~125,150

float threshold = 160;
int erodeAmt = 1;
int dilateAmt = 5;

void setup() {
  opencv = new OpenCV(this,320,240);
  size(opencv.width, opencv.height);
  noFill();

  video = new Capture(this,width,height);
  video.start();

  faceDetector = new CascadeClassifier(dataPath("haarcascade_frontalface_alt2.xml"));
  mouthDetector = new CascadeClassifier(dataPath("haarcascade_mcs_mouth.xml"));

}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);

  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  //detect face using raw Java OpenCV API
  Mat equalizedImg = opencv.getGray();
  faceDetections = new MatOfRect();
  faceDetector.detectMultiScale(equalizedImg, faceDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeFace, maxFeatureSizeFace);
  Rect[] faceDetectionResults = faceDetections.toArray();
  int faces = faceDetectionResults.length;
  text("detected faces: "+faces,5,15);
  if(faces > 0){
    Rect face = faceDetectionResults[0];
    stroke(0,192,0);
    rect(face.x,face.y,face.width,face.height);
    //detect mouth - only within face rectangle, not the whole frame
    Rect faceLower = face.clone();
    faceLower.height = (int) (face.height * 0.55);
    faceLower.y = face.y + faceLower.height; 
    //submat grabs a portion of the image (submatrix) = our region of interest (ROI)
    Mat faceROI = equalizedImg.submat(faceLower);
    Mat faceROIThresh = faceROI.clone();
    //threshold
    Imgproc.threshold(faceROI, faceROIThresh, threshold, width, Imgproc.THRESH_BINARY_INV);
    Imgproc.erode(faceROIThresh, faceROIThresh, new Mat(), new Point(-1,-1), erodeAmt);
    Imgproc.dilate(faceROIThresh, faceROIThresh, new Mat(), new Point(-1,-1), dilateAmt);
    //find contours
    Mat faceContours = faceROIThresh.clone();
    List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
    Imgproc.findContours(faceContours, contours, new Mat(), Imgproc.RETR_EXTERNAL , Imgproc.CHAIN_APPROX_SIMPLE);
    //draw contours
    for(int i = 0 ; i < contours.size(); i++){
      MatOfPoint contour = contours.get(i);
      Point[] points = contour.toArray();
      stroke(map(i,0,contours.size()-1,32,255),0,0);
      beginShape();
      for(Point p : points){
        vertex((float)p.x,(float)p.y);
      }
      endShape();
    }

    //debug view of ROI
    PImage faceImg = createImage(faceLower.width,faceLower.height,RGB);
    opencv.toPImage(faceROIThresh,faceImg);
    image(faceImg,width-faceImg.width,0);
  }
  text("Drag mouseX to control threshold: " + threshold+
      "\nHold 'e' and drag mouseX to control erodeAmt: " + erodeAmt+
      "\nHold 'd' and drag mouseX to control dilateAmt: " + dilateAmt,5,210);
}
void mouseDragged(){
  if(keyPressed){
    if(key == 'e') erodeAmt = (int)map(mouseX,0,width,1,6);
    if(key == 'd') dilateAmt = (int)map(mouseX,0,width,1,10);
  }else{
    threshold = mouseX;
  }
}
void captureEvent(Capture c) {
  c.read();
}

    可以通过使用YCrCb颜色空间更好地分割皮肤来对此进行一些改进，但是总的来说，您会注意到有很多变量需要正确设置，但这并不能使其成为非常灵活的设置。

    使用FaceOSC并通过oscP5读取处理中需要的值，您将获得更好的结果。这是FaceOSCReceiver处理示例的略微简化版本，主要侧重于嘴:
    import oscP5.*;
OscP5 oscP5;

// num faces found
int found;

// pose
float poseScale;
PVector posePosition = new PVector();


// gesture
float mouthHeight;
float mouthWidth;

void setup() {
  size(640, 480);
  frameRate(30);

  oscP5 = new OscP5(this, 8338);
  oscP5.plug(this, "found", "/found");
  oscP5.plug(this, "poseScale", "/pose/scale");
  oscP5.plug(this, "posePosition", "/pose/position");
  oscP5.plug(this, "mouthWidthReceived", "/gesture/mouth/width");
  oscP5.plug(this, "mouthHeightReceived", "/gesture/mouth/height");
}

void draw() {  
  background(255);
  stroke(0);

  if(found > 0) {
    translate(posePosition.x, posePosition.y);
    scale(poseScale);
    noFill();
    ellipse(0, 20, mouthWidth* 3, mouthHeight * 3);
  }
}

// OSC CALLBACK FUNCTIONS

public void found(int i) {
  println("found: " + i);
  found = i;
}

public void poseScale(float s) {
  println("scale: " + s);
  poseScale = s;
}

public void posePosition(float x, float y) {
  println("pose position\tX: " + x + " Y: " + y );
  posePosition.set(x, y, 0);
}

public void mouthWidthReceived(float w) {
  println("mouth Width: " + w);
  mouthWidth = w;
}

public void mouthHeightReceived(float h) {
  println("mouth height: " + h);
  mouthHeight = h;
}


// all other OSC messages end up here
void oscEvent(OscMessage m) {
  if(m.isPlugged() == false) {
    println("UNPLUGGED: " + m);
  }
}

    在OSX上，您只需下载compiled FaceOSC app即可。
    在其他操作系统上，您可能需要setup OpenFrameworks，下载ofxFaceTracker并自己编译FaceOSC。

    关于opencv - 如何在“处理”中使用网络摄像头确定上唇和下唇之间的距离？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/36883141/

    上一篇：c++ - 我如何在 opencv 中声明一个 3 维矩阵

    下一篇：Python 与 C++ OpenCV matchTemplate

    相关文章：

    matlab - 为什么 OpenCV 中的 filter2D 给出的结果与 Matlab 中的 imfilter 不同？

    c++ - OpenCV更好地检测红色？

    image - 如何判断照片是否损坏？

    java - 内部类中的处理

    java - 在处理 2.2.1 中打开 .oni 文件时出现 SimpleOpenNI 错误

    visual-studio-2010 - 错误 : opencv2/core/core_c. h:没有那个文件或目录

    python - 使用Open CV Python使用带有对角视差的StereoBM创建失灵图

    c++ - 以亚像素精度检测激光线的中心

    c++ - 使用 OpenCV 对图像产生涡旋效应

    java - 如何修复处理时棋盘游戏棋子的移动

    ©2024 IT工具网  联系我们