ios - 如何使用 AVCaptureSession 从 CMSampleBuffer 获取 UIImage

Question

我一直在尝试在 MonoTouch 中进行一些实时视频图像处理。我正在使用 AVCaptureSession 从与 AVCaptureVideoPreviewLayer 配合使用的相机获取帧。

我也成功地在我的委托(delegate)类中获得回调方法“DidOutputSampleBuffer”。但是，我尝试从生成的 CMSampleBuffer 创建 UIImage 的所有方法都失败了。

这是我设置捕获 session 的代码:

captureSession = new AVCaptureSession ();
            captureSession.BeginConfiguration ();
            videoCamera = AVCaptureDevice.DefaultDeviceWithMediaType (AVMediaType.Video);

            if (videoCamera != null)
            {
                captureSession.SessionPreset = AVCaptureSession.Preset1280x720;

                videoInput = AVCaptureDeviceInput.FromDevice (videoCamera);

                if (videoInput != null)
                    captureSession.AddInput (videoInput);

                //DispatchQueue queue = new DispatchQueue ("videoFrameQueue");

                videoCapDelegate = new videoOutputDelegate (this);

                DispatchQueue queue = new DispatchQueue("videoFrameQueue");
                videoOutput = new AVCaptureVideoDataOutput ();

                videoOutput.SetSampleBufferDelegateAndQueue (videoCapDelegate, queue);
                videoOutput.AlwaysDiscardsLateVideoFrames = true;
                videoOutput.VideoSettings.PixelFormat = CVPixelFormatType.CV24RGB;

                captureSession.AddOutput (videoOutput);

                videoOutput.ConnectionFromMediaType(AVMediaType.Video).VideoOrientation = AVCaptureVideoOrientation.Portrait;

                previewLayer = AVCaptureVideoPreviewLayer.FromSession (captureSession);
                previewLayer.Frame = UIScreen.MainScreen.Bounds;
                previewLayer.AffineTransform = CGAffineTransform.MakeRotation (Convert.DegToRad (-90));
                //this.View.Layer.AddSublayer (previewLayer);

                captureSession.CommitConfiguration ();
                captureSession.StartRunning ();
            }

我试过从样本缓冲区的图像缓冲区转换的 CVPixelBuffer 创建 CGBitmapContext，如下所示:

public override void DidOutputSampleBuffer (AVCaptureOutput captureOutput, MonoTouch.CoreMedia.CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
    {

        CVPixelBuffer pixelBuffer = sampleBuffer.GetImageBuffer () as CVPixelBuffer;
        CVReturn flag = pixelBuffer.Lock (0);
        if(flag == CVReturn.Success)
        {
            CGBitmapContext context = new CGBitmapContext
                    (
                        pixelBuffer.BaseAddress,
                        pixelBuffer.Width,
                        pixelBuffer.Height,
                        8,
                        pixelBuffer.BytesPerRow, 
                        CGColorSpace.CreateDeviceRGB (), 
                        CGImageAlphaInfo.PremultipliedFirst
                        );

            UIImage image = new UIImage(context.ToImage());

            ProcessImage (image);

            pixelBuffer.Unlock(0);

        }else
            Debug.Print(flag.ToString()

        sampleBuffer.Dispose();
    }

这会导致以下错误

<Error>: CGBitmapContextCreate: invalid data bytes/row: should be at least 2880 for 8 integer bits/component, 3 components, kCGImageAlphaPremultipliedFirst.

即使对参数进行了一些调整，我还是会收到无效的 Handle 异常或 native objective-c 中的段错误。

我也试过简单地使用 CVImageBuffer 创建一个 CIImage 并从中创建一个 UIImage，如下所示:

public override void DidOutputSampleBuffer (AVCaptureOutput captureOutput, MonoTouch.CoreMedia.CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
    {

        CIImage cImage = new CIImage(sampleBuffer.GetImageBuffer ());
        UIImage image = new UIImage(cImage);
        ProcessImage (image);

        sampleBuffer.Dispose();
    }

这会导致在初始化 CIImage 时出现异常:

NSInvalidArgumentException Reason: -[CIImage initWithCVImageBuffer:]: unrecognized selector sent to instance 0xc821d0

老实说，这感觉像是 MonoTouch 的某种错误，但如果我遗漏了什么或者只是想以一种奇怪的方式来做这件事，请让我知道一些替代解决方案。

谢谢

Answer 1

此错误消息对此进行了解释:

<Error>: CGBitmapContextCreate: invalid data bytes/row: should be at least 2880 for 8 integer bits/component, 3 components, kCGImageAlphaPremultipliedFirst.

宽度为 720 像素，每行 1084 字节，即每像素 1.5 字节多一点 - 这不是 RGB24(每像素 3 字节)，它是某种平面格式。

您可能需要检查 AVCaptureVideoDataOutput.AvailableVideoCVPixelFormatTypes 以了解您可以使用的可用像素格式，看看是否有任何支持的格式更容易使用。