After many factory resets, I have managed to make it work with just forgetting the iPhone in my MAC. After this, I had no issue with the AirPods in my MAC (Big Sur 11.0.1)