Async AVAudioPlayerNode.scheduleBuffer stutters

My code that streams buffers into AVAudioPlayerNode is stuttering when the buffer is finished and before the next one is played.

while engine.isRunning {
    let framesToCopy = min(buffer.frameLength - framePosition, Self.BufferSize)
    let srcRaw = UnsafeRawPointer(srcPtr)
                
    let playbackBuffer = AVAudioPCMBuffer(pcmFormat: buffer.format, frameCapacity: Self.BufferSize)!
    let playbackPtr = playbackBuffer.floatChannelData![0]
    let destRaw = UnsafeMutableRawPointer(mutating: playbackPtr)

    memcpy(destRaw, srcRaw, Int(framesToCopy) * MemoryLayout<Float>.stride)
                
    srcPtr = srcPtr.advanced(by: Int(framesToCopy))
    playbackBuffer.frameLength = framesToCopy
                
    await player.scheduleBuffer(playbackBuffer,
                                            at: nil,
                                            options: [],
                                            completionCallbackType: .dataRendered)
}

I've tried to schedule multiple buffers at once using a combination of both the synchronous and async versions of scheduleBuffer because I thought the delay might be but it still stutters and the data copied into the playbackBuffer matches the source buffer. I've tried all combinations of options and completionCallbackType but no luck.

I've tried increasing the buffer size but that just spaces out the stutters because the buffer is larger.

What am I missing about this API?