iOS 10.0 - 10.1 : AVPlayerLayer doesn't show video after using AVVideoCompositionCoreAnimationTool, 只有音频

Question

如果您想自己运行，这里有一个完整的项目:https://www.dropbox.com/s/5p384mogjzflvqk/AVPlayerLayerSoundOnlyBug_iOS10.zip?dl=0

这是 iOS 10 上的新问题，从 iOS 10.2 开始已修复。使用 AVAssetExportSession 和 AVVideoCompositionCoreAnimationTool 导出视频后，在导出过程中在视频顶部合成图层后，在 AVPlayerLayer 中播放的视频无法播放。这似乎不是由达到 AV 编码/解码管道限制引起的，因为它经常发生在单次导出之后，据我所知，这只会启动 2 个管道:1 个用于 AVAssetExportSession，另一个用于 AVPlayer。我还正确地设置了图层的框架，正如您通过运行下面的代码看到的那样，它为图层提供了一个您可以清楚地看到的蓝色背景。

导出后，在播放视频之前等待一段时间似乎会使其更加可靠，但这并不是告诉用户的真正可接受的解决方法。

关于导致此问题的原因或我如何解决或解决它的任何想法？我是否搞砸了某些事情或遗漏了重要的步骤或细节？非常感谢任何帮助或文档指针。

import UIKit
import AVFoundation

/* After exporting an AVAsset using AVAssetExportSession with AVVideoCompositionCoreAnimationTool, we
 * will attempt to play a video using an AVPlayerLayer with a blue background.
 *
 * If you see the blue background and hear audio you're experiencing the missing-video bug. Otherwise
 * try hitting the button again.
 */

class ViewController: UIViewController {
    private var playerLayer: AVPlayerLayer?
    private let button = UIButton()
    private let indicator = UIActivityIndicatorView(activityIndicatorStyle: .gray)

    override func viewDidLoad() {
        super.viewDidLoad()
        view.backgroundColor = UIColor.white
        button.setTitle("Cause Trouble", for: .normal)
        button.setTitleColor(UIColor.black, for: .normal)
        button.addTarget(self, action: #selector(ViewController.buttonTapped), for: .touchUpInside)
        view.addSubview(button)
        button.translatesAutoresizingMaskIntoConstraints = false
        NSLayoutConstraint.activate([
            button.centerXAnchor.constraint(equalTo: view.centerXAnchor),
            button.bottomAnchor.constraint(equalTo: view.bottomAnchor, constant: -16),
        ])

        indicator.hidesWhenStopped = true
        view.insertSubview(indicator, belowSubview: button)
        indicator.translatesAutoresizingMaskIntoConstraints = false
        NSLayoutConstraint.activate([
            indicator.centerXAnchor.constraint(equalTo: button.centerXAnchor),
            indicator.centerYAnchor.constraint(equalTo: button.centerYAnchor),
        ])
    }

    func buttonTapped() {
        button.isHidden = true
        indicator.startAnimating()
        playerLayer?.removeFromSuperlayer()

        let sourcePath = Bundle.main.path(forResource: "video.mov", ofType: nil)!
        let sourceURL = URL(fileURLWithPath: sourcePath)
        let sourceAsset = AVURLAsset(url: sourceURL)

        //////////////////////////////////////////////////////////////////////
        // STEP 1: Export a video using AVVideoCompositionCoreAnimationTool //
        //////////////////////////////////////////////////////////////////////
        let exportSession = { () -> AVAssetExportSession in
            let sourceTrack = sourceAsset.tracks(withMediaType: AVMediaTypeVideo).first!

            let parentLayer = CALayer()
            parentLayer.frame = CGRect(origin: .zero, size: CGSize(width: 1280, height: 720))
            let videoLayer = CALayer()
            videoLayer.frame = parentLayer.bounds
            parentLayer.addSublayer(videoLayer)

            let composition = AVMutableVideoComposition(propertiesOf: sourceAsset)
            composition.animationTool = AVVideoCompositionCoreAnimationTool(postProcessingAsVideoLayer: videoLayer, in: parentLayer)
            let layerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: sourceTrack)
            layerInstruction.setTransform(sourceTrack.preferredTransform, at: kCMTimeZero)
            let instruction = AVMutableVideoCompositionInstruction()
            instruction.timeRange = CMTimeRange(start: kCMTimeZero, duration: sourceAsset.duration)
            instruction.layerInstructions = [layerInstruction]
            composition.instructions = [instruction]

            let e = AVAssetExportSession(asset: sourceAsset, presetName: AVAssetExportPreset1280x720)!
            e.videoComposition = composition
            e.outputFileType = AVFileTypeQuickTimeMovie
            e.timeRange = CMTimeRange(start: kCMTimeZero, duration: sourceAsset.duration)
            let outputURL = URL(fileURLWithPath: NSTemporaryDirectory().appending("/out2.mov"))
            _ = try? FileManager.default.removeItem(at: outputURL)
            e.outputURL = outputURL
            return e
        }()

        print("Exporting asset...")
        exportSession.exportAsynchronously {
            assert(exportSession.status == .completed)

            //////////////////////////////////////////////
            // STEP 2: Play a video in an AVPlayerLayer //
            //////////////////////////////////////////////
            DispatchQueue.main.async {
                // Reuse player layer, shouldn't be hitting the AV pipeline limit
                let playerItem = AVPlayerItem(asset: sourceAsset)
                let layer = self.playerLayer ?? AVPlayerLayer()
                if layer.player == nil {
                    layer.player = AVPlayer(playerItem: playerItem)
                }
                else {
                    layer.player?.replaceCurrentItem(with: playerItem)
                }
                layer.backgroundColor = UIColor.blue.cgColor
                if UIDeviceOrientationIsPortrait(UIDevice.current.orientation) {
                    layer.frame = self.view.bounds
                    layer.bounds.size.height = layer.bounds.width * 9.0 / 16.0
                }
                else {
                    layer.frame = self.view.bounds.insetBy(dx: 0, dy: 60)
                    layer.bounds.size.width = layer.bounds.height * 16.0 / 9.0
                }
                self.view.layer.insertSublayer(layer, at: 0)
                self.playerLayer = layer

                layer.player?.play()
                print("Playing a video in an AVPlayerLayer...")

                self.button.isHidden = false
                self.indicator.stopAnimating()
            }
        }
    }
}

Answer 1

在这种情况下，我的答案是解决 AVVideoCompositionCoreAnimationTool 的问题。通过使用实现 AVVideoCompositing 的自定义视频合成类协议(protocol)，以及实现 AVVideoCompositionInstruction 的自定义组合指令协议(protocol)。因为我需要覆盖 CALayer在视频之上，我将该层包含在合成指令实例中。

您需要像这样在您的视频合成上设置自定义合成器:

composition.customVideoCompositorClass = CustomVideoCompositor.self

然后在上面设置你的自定义指令:

let instruction = CustomVideoCompositionInstruction(...) // whatever parameters you need and are required by the instruction protocol
composition.instructions = [instruction]

编辑:这是一个工作示例，说明如何使用自定义合成器通过 GPU 在视频上叠加图层:https://github.com/samsonjs/LayerVideoCompositor ... 原始答案在下面继续

至于合成器本身，如果您观看相关的 WWDC session 并查看它们的示例代码，您可以实现一个。我不能在这里发布我写的那个，但我正在使用 CoreImage 来处理 AVAsynchronousVideoCompositionRequest 的繁重工作。 ，确保使用 OpenGL CoreImage 上下文以获得最佳性能(如果你在 CPU 上执行它将会非常慢)。如果在导出期间出现内存使用高峰，您可能还需要一个自动释放池。

如果您要覆盖 CALayer像我一样确保设置 layer.isGeometryFlipped = true当您将该层渲染到 CGImage 时在将其发送到 CoreImage 之前。并确保缓存呈现的 CGImage在合成器中从一帧到另一帧。