ios - 如何为YCbCr图像创建vImage_CGImageFormat？

Question

我想创建一个库，可以将使用 AVCaptureDevice 捕获的任何视频帧转换为目标像素格式。因此，它必须支持 kCVPixelFormatType_32BGRA、kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange、kCVPixelFormatType_420YpCbCr8BiPlanarFullRange 以及可选的 kCVPixelFormatType_420YpCbCr8Planar。

为了提高性能，我想使用 Accelerate 框架，因为它包含丰富的 conversion functions 集。 。由于目标像素格式可能不同并且由库用户设置，因此很高兴使用通用 vImageConvert_AnyToAny功能:

You use vImage's vImageConvert_AnyToAny(::::_:) function to convert between Core Video or Core Graphics image data of arbitrary color spaces and bit depths. The source and destination images are described by one or more buffers. For example, a Y'CbCr image may be composed of one buffer containing luminance information and one buffer containing chrominance information.

要使用此函数，我必须创建 vImageConverter定义图像之间的转换。此类的构造函数需要以 vImage_CGImageFormat 形式描述源和目标图像格式描述。 :

vImage_CGImageFormat in_format = ...;
vImage_CGImageFormat out_format = ...; 
vImageConverterRef converter = vImageConverter_CreateWithCGImageFormat(&in_format, &out_format, NULL, kvImagePrintDiagnosticsToConsole, &err);

if( err == kvImageNoError )
{
    vImage_Buffer *src_planes = ...;
    vImage_Buffer *dst_planes = ...;

    err = vImageConvert_AnyToAny(converter, src_planes, dst_planes, NULL,  kvImagePrintDiagnosticsToConsole);
}

代码基于这篇 Apple 文章:Building a Basic Conversion Workflow

对于kCVPixelFormatType_32BGRA，这样的vImage_CGImageFormat很简单，并在vImage_Utilities.h中进行了描述:

/*!
 * @struct vImage_CGImageFormat
 * @abstract A pixel format
 * @discussion A vImage_CGImageFormat describes the ordering of the color channels, how many there are,
 * the size and type of the data in the color channels and whether the data is premultiplied by alpha or not.
 * This format mirrors the image format descriptors used by CoreGraphics to create things like CGImageRef and
 * CGBitmapContextRef.
 *
 * This vImage_CGImageFormat:
 *
 *  <pre>@textblock
 *      vImage_CGImageFormat format = {
 *          .bitsPerComponent = 8,
 *          .bitsPerPixel = 32,
 *          .colorSpace = CGColorSpaceCreateDeviceRGB(),                                    // don't forget to release this!
 *          .bitmapInfo = kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little,
 *          .version = 0,                                                                   // must be 0
 *          .decode = NULL,
 *          .renderingIntent = kCGRenderingIntentDefault
 *      };
 *  @/textblock</pre>
 *
 * codes for a little endian ARGB8888 pixel, or what is called in the rest of vImage, BGRA8888. Note: for 16-
 * and 32-bits per component formats (int16_t, uint16_t, half-float, float) most vImage image filters assume
 * the data is in host-endian format. (The APIs in this header do not.) Host-endian is little endian for Intel
 * and ARM, big endian for PowerPC. If the data is not in host-endian format, then you may use
 * vImagePermuteChannels_ARGB8888 or vImageByteSwap_Planar16U to swap the image data byte ordering.
 *
 * Some examples:
 *  <pre>@textblock
 *      ARGB8888     ->  {8, 32, NULL, alpha first, 0, NULL, kCGRenderingIntentDefault}     alpha first = { kCGImageAlphaFirst, kCGImageAlphaPremultipliedFirst, kCGImageAlphaNoneSkipFirst }
 *      RGBA8888     ->  {8, 32, NULL, alpha last,  0, NULL, kCGRenderingIntentDefault}     alpha last  = { kCGImageAlphaLast,  kCGImageAlphaPremultipliedLast,  kCGImageAlphaNoneSkipLast }
 *      BGRA8888     ->  {8, 32, NULL, alpha first | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault}
 *      RGB888       ->  {8, 24, NULL, kCGImageAlphaNone | kCGBitmapByteOrderDefault, 0, NULL, kCGRenderingIntentDefault}
 *      RGB565       ->  {5, 16, NULL, kCGImageAlphaNone | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
 *      ARGB1555     ->  {5, 16, NULL, alpha first | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
 *      RGBA16F      ->  {16, 64, NULL, alpha last | kCGBitmapFloatComponents | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault }
 *      CMYK8888     ->  {8, 32, CGColorSpaceCreateDeviceCMYK(), kCGImageAlphaNone, 0, NULL, kCGRenderingIntentDefault  }
 *      ARGBFFFF premultiplied    ->  {32, 128, NULL, kCGImageAlphaPremultipliedFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *      ARGBFFFF not-premultiplied -> {32, 128, NULL, kCGImageAlphaFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *      ARGBFFFF, alpha = 1 ->        {32, 128, NULL, kCGImageAlphaNoneSkipFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *  @/textblock</pre>
 *
 *  Note that some of these formats, particularly RGB565 and 16F formats are supported by vImage but
 *  not necessarily CoreGraphics. They will be converted to a higher precision format as necessary by
 *  vImage in vImageCreateCGImageFromBuffer().
 *
 *  By C rules, uninitialized struct parameters are set to zero. The last three parameters are usually zero, so can usually be omitted.
 *
 *  <pre>@textblock
 *      vImage_CGImageFormat srgb888 = (vImage_CGImageFormat){
 *          .bitsPerComponent = 8,
 *          .bitsPerPixel = 24,
 *          .colorSpace = NULL,
 *          .bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault };
 *  @/textblock</pre>
 *
 * To understand how these various parameters relate to one another, we can look at the process of converting from
 * one vImage_CGImageFormat format to another:
 *
 *  1) transform endianness of src format given by bitmapInfo to host endian  (except 8 bitPerComponent content)
 *  2) remove decode array transformation, and up convert to a higher range format as necessary to preserve precision / range
 *  3) convert src colorspace to reference XYZ colorspace (may cause upconvert to preserve range / precision)
 *  4) convert XYZ to destination colorspace + rendering intent
 *  5) convert to destination precision (given by bitsPerComponent)
 *  6) deal with any alpha changes (given by bitmapInfo) or flattening that needs to occur
 *  7) Apply any channel reordering requested, if it didn't happen at an earlier step. (As indicated by src and dest bitmapInfo)
 *  8) Apply destination decode array
 *  9) Apply endianness transform given by dest bitmapInfo
 *
 * Clearly, for most common transformations not all steps need to occur and multiple steps can be collapsed into a compound operation.
 *
 *  @field  bitsPerComponent    The number of bits needed to represent one channel of data in one pixel. For ARGB8888, this would be 8. Expected values: {1, 2, 4, 5, 8, 10, 12, 16, 32}
 *  @field  bitsPerPixel        The number of bits needed to represent one pixel. For ARGB8888, this would be 32.
 *                              It is possible that bitsPerPixel > bitsPerComponent * number of components, but in practice this is rare.
 *                              The number of color components is given by the colorspace and the number of alpha components (0 or 1) is given by
 *                              by the bitmapInfo.
 *  @field  colorSpace          A description of how the pixel data in the image is positioned relative to a reference XYZ color space.
 *                                  See CoreGraphics/CGColorSpace.h.  Pass NULL as a shorthand for sRGB. The vImage_CGImageFormat is not
 *                                  capable of managing the memory held by the colorSpace. If you created the colorspace, you must
 *                                  be sure to release it before all references to it disappear from scope.
 *  @field  bitmapInfo          The CGBitmapInfo describing the color channels. See CoreGraphics/CGImage.h.
 *                                  ARGB8888 is kCGImageAlphaFirst | kCGBitmapByteOrderDefault
 *                                  BGRA8888 is kCGImageAlphaFirst | kCGBitmapByteOrder32Little
 *  @field  version             The struct is versioned for future expansion.  Pass 0 here.
 *  @field  decode              Prior to transformations caused by the colorspace, color channels are subject to a linear transformation.
 *                              This allows for a different range than the typical [0,1.0]. NULL indicates default behavior of [0,1.0]
 *                              range, and is what you should use if you don't understand this parameter. See description of CGImageCreate()
 *                              for a discussion of decode arrays. See also Decode Arrays section of Chapter 4.8 of the PDF specification.
 *                              The vImage_CGImageFormat is not capable of managing the memory held by the decode array. If you created a
 *                              decode array on the heap, you must be sure to release it before all references to it disappear from scope.
 *
 *  @field renderingIntent      See CGColorSpace.h. kCGRenderingIntentDefault is typical here. By convention, rendering intent changes that
 *                              are not accompanied by a colorspace change are ignored.
 */

我不明白如何为 YCbCr 像素格式创建 vImage_CGImageFormat。

首先我认为它根本不支持，但是这种格式有特殊的函数:Understanding YpCbCr Image Formats vImageConverter 对于多平面图像有特殊功能，例如 vImageConverter_GetNumberOfSourceBuffers和 vImageConverter_GetSourceBufferOrder 。最后一个函数中有很多vImage Buffer Type Codes甚至kvImageBufferTypeCode_CVPixelBuffer_YCbCr。

所以，看起来可以为 YCbCr 创建 vImageConverter，我需要帮助来了解如何做到这一点。

Answer 1

vImageConverter_CreateWithCGImageFormat创建一个用于在两种 Core Graphics 格式之间进行转换的转换器。

在这种情况下，您不应该使用 vImageConverter_CreateForCVToCGImageFormat从 CVPixelBuffer 转换(使用 vImageCVImageFormat 可以使用 vImageCVImageFormat_CreateWithCVPixelBuffer 生成)到 CGImage ？ vImage_CGImageFormat是您的目标格式 - 因此其属性由您定义。