ios - 将音频从 Watch 流式传输到 iPhone 以使用 SFSpeechRecognizer

Question

我想在我的 Watch 应用中进行语音识别，显示实时转录。由于 SFSpeechRecognizer 在 WatchOS 上不可用，因此我使用 WatchConnectivity 将应用设置为将音频流式传输到 iOS 配套设备。在尝试此操作之前，我在 iPhone 上尝试了相同的操作，相同的代码，但不涉及 watch - 它可以在那里工作。

通过我的流式传输尝试，同伴将接收音频 block 并且不会抛出任何错误，但它也不会转录任何文本。我怀疑在从 转换时我做错了什么AVAudioPCMBuffer 和回来，但我不能完全确定它，因为我缺乏使用原始数据和指针的经验。

现在，整个过程如下:

1. 用户按下按钮，触发 Watch 要求 iPhone 设置recognitionTask
2. iPhone 设置 recognitionTask 并回答“正常”或出现一些错误:

guard let speechRecognizer = self.speechRecognizer else {
    WCManager.shared.sendWatchMessage(.speechRecognitionRequest(.error("no speech recognizer")))
    return
}
recognitionRequest = SFSpeechAudioBufferRecognitionRequest()
guard let recognitionRequest = recognitionRequest else {
    WCManager.shared.sendWatchMessage(.speechRecognitionRequest(.error("speech recognition request denied by ios")))
    return
}
recognitionRequest.shouldReportPartialResults = true
if #available(iOS 13, *) {
    recognitionRequest.requiresOnDeviceRecognition = true
}

recognitionTask = speechRecognizer.recognitionTask(with: recognitionRequest) { result, error in
    if let result = result {
        let t = result.bestTranscription.formattedString
        WCManager.shared.sendWatchMessage(.recognizedSpeech(t))
    }
    
    if error != nil {
        self.recognitionRequest = nil
        self.recognitionTask = nil
        WCManager.shared.sendWatchMessage(.speechRecognition(.error("?")))
    }
}
WCManager.shared.sendWatchMessage(.speechRecognitionRequest(.ok))

Watch 设置 Audio Session ，在音频引擎的输入节点上安装一个 Tap，并将音频格式返回到 iPhone:

do {
    try startAudioSession()
} catch {
    self.state = .error("couldn't start audio session")
    return
}

let inputNode = audioEngine.inputNode
let recordingFormat = inputNode.outputFormat(forBus: 0)
inputNode.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat)
    { (buffer: AVAudioPCMBuffer, when: AVAudioTime) in
        let audioBuffer = buffer.audioBufferList.pointee.mBuffers
        let data = Data(bytes: audioBuffer.mData!, count: Int(audioBuffer.mDataByteSize))
        if self.state == .running {
            WCManager.shared.sendWatchMessage(.speechRecognition(.chunk(data, frameCount: Int(buffer.frameLength))))
        }
    }
audioEngine.prepare()

do {
    let data = try NSKeyedArchiver.archivedData(withRootObject: recordingFormat, requiringSecureCoding: true)
    WCManager.shared.sendWatchMessage(.speechRecognition(.audioFormat(data)),
        errorHandler: { _ in
            self.state = .error("iphone unavailable")
    })
    self.state = .sentAudioFormat
} catch {
    self.state = .error("could not convert audio format")
}

iPhone 保存音频格式并返回 .ok 或 .error():

guard let format = try? NSKeyedUnarchiver.unarchivedObject(ofClass: AVAudioFormat.self, from: data) else {
    // ...send back .error, destroy the recognitionTask
}
self.audioFormat = format
// ...send back .ok

watch 启动音频引擎

try audioEngine.start()

iPhone 接收音频 block 并将其附加到 recognitionRequest:

guard let pcm = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: AVAudioFrameCount(frameCount)) else {
    // ...send back .error, destroy the recognitionTask
}

let channels = UnsafeBufferPointer(start: pcm.floatChannelData, count: Int(pcm.format.channelCount))
let data = chunk as NSData
data.getBytes(UnsafeMutableRawPointer(channels[0]), length: data.length)
recognitionRequest.append(pcm)

任何想法都将受到高度赞赏。感谢您抽出时间!

Answer 1

复制内存后我忘记更新AVAudioPCMBuffer.frameLength。现在可以完美运行，没有任何明显的延迟:)

// ...
data.getBytes(UnsafeMutableRawPointer(channels[0]), length: data.length)
pcm.frameLength = AVAudioFrameCount(frameCount)
// ...