I have the same problem. I have very fast optic fiber connection, so it's not the problem.  I did an NVRAM restart (cmd+opt+P+R) - and it solved the issue but for how long? Mac mini (late 2014) 16 gigs memory. Big Sur 11.3