I'm experiencing the exact same issue on Ventura / tvOS 16.4. It's works with 1st Siri Remote but not with well with the 2nd generation.