opengl-es - 使用 CVPixelBufferRef 和 Shaders 在 ffmpeg 的 OpenGL 中渲染 YUV 视频

Question

我正在使用 iOS 5.0 方法“CVOpenGLESTextureCacheCreateTextureFromImage”来渲染 ffmpeg 的 YUV 帧。

我正在使用苹果示例 GLCameraRipple

我在 iPhone 屏幕上的结果是:iPhone Screen

我需要知道我做错了。

我放了部分代码来查找错误。

ffmpeg 配置帧:

ctx->p_sws_ctx = sws_getContext(ctx->p_video_ctx->width, 
                                ctx->p_video_ctx->height, 
                                ctx->p_video_ctx->pix_fmt, 
                                ctx->p_video_ctx->width, 
                                ctx->p_video_ctx->height,
                                PIX_FMT_YUV420P, SWS_FAST_BILINEAR, NULL, NULL, NULL);


// Framebuffer for RGB data
ctx->p_frame_buffer = malloc(avpicture_get_size(PIX_FMT_YUV420P,
                                                ctx->p_video_ctx->width, 
                                                ctx->p_video_ctx->height));

avpicture_fill((AVPicture*)ctx->p_picture_rgb, ctx->p_frame_buffer,PIX_FMT_YUV420P, 
               ctx->p_video_ctx->width, 
               ctx->p_video_ctx->height);

我的渲染方法:

if (NULL == videoTextureCache) {
    NSLog(@"displayPixelBuffer error");
    return;
}    


CVPixelBufferRef pixelBuffer;    
   CVPixelBufferCreateWithBytes(kCFAllocatorDefault, mTexW, mTexH, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, buffer, mFrameW * 3, NULL, 0, NULL, &pixelBuffer);



CVReturn err;    
// Y-plane
glActiveTexture(GL_TEXTURE0);
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, 
                                                   videoTextureCache,
                                                   pixelBuffer,
                                                   NULL,
                                                   GL_TEXTURE_2D,
                                                   GL_RED_EXT,
                                                   mTexW,
                                                   mTexH,
                                                   GL_RED_EXT,
                                                   GL_UNSIGNED_BYTE,
                                                   0,
                                                   &_lumaTexture);
if (err) 
{
    NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
}   

glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);     

// UV-plane
glActiveTexture(GL_TEXTURE1);
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, 
                                                   videoTextureCache,
                                                   pixelBuffer,
                                                   NULL,
                                                   GL_TEXTURE_2D,
                                                   GL_RG_EXT,
                                                   mTexW/2,
                                                   mTexH/2,
                                                   GL_RG_EXT,
                                                   GL_UNSIGNED_BYTE,
                                                   1,
                                                   &_chromaTexture);
if (err) 
{
    NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
}

glBindTexture(CVOpenGLESTextureGetTarget(_chromaTexture), CVOpenGLESTextureGetName(_chromaTexture));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);     

glBindFramebuffer(GL_FRAMEBUFFER, defaultFramebuffer);

// Set the view port to the entire view
glViewport(0, 0, backingWidth, backingHeight);

static const GLfloat squareVertices[] = {
    1.0f, 1.0f,
    -1.0f, 1.0f,
    1.0f,  -1.0f,
    -1.0f,  -1.0f,
};

GLfloat textureVertices[] = {
    1, 1,
    1, 0,
    0, 1,
    0, 0,
};

// Draw the texture on the screen with OpenGL ES 2
[self renderWithSquareVertices:squareVertices textureVertices:textureVertices];


// Flush the CVOpenGLESTexture cache and release the texture
CVOpenGLESTextureCacheFlush(videoTextureCache, 0);    
CVPixelBufferRelease(pixelBuffer);     

 [moviePlayerDelegate bufferDone];

RenderWithSquareVertices 方法

    - (void)renderWithSquareVertices:(const GLfloat*)squareVertices textureVertices:(const GLfloat*)textureVertices
{


  // Use shader program.
    glUseProgram(shader.program);

// Update attribute values.
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, squareVertices);
glEnableVertexAttribArray(ATTRIB_VERTEX);
glVertexAttribPointer(ATTRIB_TEXTUREPOSITON, 2, GL_FLOAT, 0, 0, textureVertices);
glEnableVertexAttribArray(ATTRIB_TEXTUREPOSITON);

glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

// Present
glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);
[context presentRenderbuffer:GL_RENDERBUFFER];

}

我的片段着色器:

uniform sampler2D SamplerY;
uniform sampler2D SamplerUV;


varying highp vec2 _texcoord;

void main()
{

mediump vec3 yuv;
lowp vec3 rgb;

yuv.x = texture2D(SamplerY, _texcoord).r;
yuv.yz = texture2D(SamplerUV, _texcoord).rg - vec2(0.5, 0.5);

// BT.601, which is the standard for SDTV is provided as a reference

/* rgb = mat3(    1,       1,     1,
 0, -.34413, 1.772,
 1.402, -.71414,     0) * yuv;*/


// Using BT.709 which is the standard for HDTV
rgb = mat3(      1,       1,      1,
           0, -.18732, 1.8556,
           1.57481, -.46813,      0) * yuv;

   gl_FragColor = vec4(rgb, 1);

}

很感谢，

Answer 1

我想问题在于 YUV420(或 I420)是一种三平面图像格式。 I420 是一个 8 位 Y 平面，后跟 8 位 2x2 二次采样 U 和 V 平面。来自 GLCameraRipple 的代码需要 NV12 格式:8 位 Y 平面，后跟具有 2x2 子采样的交错 U/V 平面。鉴于此，我预计您将需要三个纹理。 luma_tex、u_chroma_tex、v_chroma_tex。

另请注意，GLCameraRipple 也可能期待“视频范围”。换句话说，平面格式的值是 luma=[16,235] chroma=[16,240]。