Oh I'm glad I'm not alone. I left it downloading since yesterday and it's still downloading. Haha!

You're not alone. This problem only occured when I upgraded to Big Sur. What I noticed is that I can now connect to 5ghz networks easily. I'm now having a hard time connecting via 2.4Ghz, using the same SSID. Once again, this only happened after upgrading to Big Sur, from Catalina.