I've somewhat figured this out. I can get skip to work but I still haven't managed to get play/pause from the keyboard to work.
The thing that I was missing is on macOS you need to update the playing state on MPNowPlayingInfoCenter. It can't just infer it from the AVAudioSession like it does on other platforms.
.i.e.
MPNowPlayingInfoCenter.default().playbackState = .playing
Since I started doing that my app shows up in 'Now Playing' and all the controls work fine there. I just can't figure out why play/pause on the keyboard still doesn't work.