swift - 使用 opencv : UIImage too slow 实时处理 ARkit 帧

Question

我正在尝试在 ios 上使用 swift 和 Objective-C++ 开发一个带有 ARkit 和 OpenCV 的增强现实应用程序。我需要使用 opencv 实时处理 ARkit 视频帧(以便用我转换后的视频替换视频)。

为此，我使用了 frame.capturedImage，它是一个 CVPixelBuffer，我将其转换为 cv::mat。然后，我制作我的 opencv 东西，并将结果转换为 UIImage(下面的代码)。

它可以工作，但速度非常非常慢(低于 2 FPS)。我是 IOS 编程的新手，但我认为使用 UIImage 不是正确的方法......

你有想法吗？我可以做些什么来优化我的应用程序？提前谢谢你...

这是我的代码:

从缓冲区到 cv::mat 的转换:

+ (Mat)_matFromBuffer:(CVImageBufferRef)buffer {

    Mat mat;
    CVPixelBufferLockBaseAddress(buffer,0);
    //Get the data from the first plane (Y)
    void *address = CVPixelBufferGetBaseAddressOfPlane(buffer, 0);
    int bufferWidth = (int)CVPixelBufferGetWidthOfPlane(buffer,0);
    int bufferHeight = (int)CVPixelBufferGetHeightOfPlane(buffer, 0);
    int bytePerRow = (int)CVPixelBufferGetBytesPerRowOfPlane(buffer, 0);
    //Get the pixel format
    OSType pixelFormat = CVPixelBufferGetPixelFormatType(buffer);

    Mat converted;
    //NOTE: CV_8UC3 means unsigned (0-255) 8 bits per pixel, with 3 channels!
    //Check to see if this is the correct pixel format
    if (pixelFormat == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
        //We have an ARKIT buffer
        //Get the yPlane (Luma values)
        Mat yPlane = Mat(bufferHeight, bufferWidth, CV_8UC1, address);
        //Get cbcrPlane (Chroma values)
        int cbcrWidth = (int)CVPixelBufferGetWidthOfPlane(buffer,1);
        int cbcrHeight = (int)CVPixelBufferGetHeightOfPlane(buffer, 1);
        void *cbcrAddress = CVPixelBufferGetBaseAddressOfPlane(buffer, 1);
        //Since the CbCr Values are alternating we have 2 channels: Cb and Cr. Thus we need to use CV_8UC2 here.
        Mat cbcrPlane = Mat(cbcrHeight, cbcrWidth, CV_8UC2, cbcrAddress);
        //Split them apart so we can merge them with the luma values
        vector<Mat> cbcrPlanes;
        split(cbcrPlane, cbcrPlanes);

        Mat cbPlane;
        Mat crPlane;
        //Since we have a 4:2:0 format, cb and cr values are only present for each 2x2 luma pixels. Thus we need to enlargen them (by a factor of 2).
        resize(cbcrPlanes[0], cbPlane, yPlane.size(), 0, 0, INTER_NEAREST);
        resize(cbcrPlanes[1], crPlane, yPlane.size(), 0, 0, INTER_NEAREST);

        Mat ycbcr;
        vector<Mat> allPlanes = {yPlane, cbPlane, crPlane};
        merge(allPlanes, ycbcr);
        //ycbcr now contains all three planes. We need to convert it from YCbCr to RGB so OpenCV can work with it
        cvtColor(ycbcr, converted, COLOR_YCrCb2RGB);
    } else {
        //Probably RGB so just use that.
        converted = Mat(bufferHeight, bufferWidth, CV_8UC3, address, bytePerRow).clone();
    }

    //Since we clone the cv::Mat no need to keep the Buffer Locked while we work on it.
    CVPixelBufferUnlockBaseAddress(buffer, 0);

    Mat rotated;
    transpose(converted, rotated);
    flip(rotated,rotated, 1);

    return rotated;
}

从 mat 到 UIImage 的转换:

+ (UIImage *)_imageFrom:(Mat)source {
    cout << "-> imageFrom\n";

    NSData *data = [NSData dataWithBytes:source.data length:source.elemSize() * source.total()];
    CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data);

    CGBitmapInfo bitmapFlags = kCGImageAlphaNone | kCGBitmapByteOrderDefault;
    size_t bitsPerComponent = 8;
    size_t bytesPerRow = source.step[0];
    CGColorSpaceRef colorSpace = (source.elemSize() == 1 ? CGColorSpaceCreateDeviceGray() : CGColorSpaceCreateDeviceRGB());

    CGImageRef image = CGImageCreate(source.cols, source.rows, bitsPerComponent, bitsPerComponent * source.elemSize(), bytesPerRow, colorSpace, bitmapFlags, provider, NULL, false, kCGRenderingIntentDefault);
    UIImage *result = [UIImage imageWithCGImage:image];

    CGImageRelease(image);
    CGDataProviderRelease(provider);
    CGColorSpaceRelease(colorSpace);

    return result;
}

在 ViewController.swift 中

func session(_ session: ARSession, didUpdate frame: ARFrame) {
        let openCVWrapper = OpenCVWrapper()
        openCVWrapper.isThisWorking()

        // Present an error message to the user
        let pixelBuffer = frame.capturedImage
        if (CVPixelBufferGetPlaneCount(pixelBuffer) < 2) {
            return
        }

        // works but too slow (2 FPS)
        let bckgd = OpenCVWrapper.opencvstuff(pixelBuffer) as UIImage!;
        sceneView.scene.background.contents = bckgd;  
    }

Answer 1

我找到了一个使用 Objective-C 的解决方案，我认为您可能希望自己将其转换为 swift。

Draw a circle in capturedImage iOS - ARKit Ycbcr

AR 框架。 capturedImage 是 Ycbcr 缓冲区。来自jgh的回答How seperate y-planar, u-planar and uv-planar from yuv bi planar in ios?

The Y plane represents the luminance component, and the UV plane represents the Cb and Cr chroma components.

In the case of kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format, you will find the luma plane is 8bpp with the same dimensions as your video, your chroma plane will be 16bpp, but only a quarter of the size of the original video. You will have one Cb and one Cr component per pixel on this plane.

so if your input video is 352x288, your Y plane will be 352x288 8bpp, and your CbCr 176x144 16bpp. This works out to be about the same amount of data as a 12bpp 352x288 image, half what would be required for RGB888 and still less than RGB565.

So in the buffer, Y will look like this [YYYYY . . . ] and UV [UVUVUVUVUV . . .]

vs RGB being, of course, [RGBRGBRGB . . . ]

我们应该在 cv::Mat 中使用 yPlane 和 cbcrPlane 进行图像处理，查看我的代码了解更多详情。