I’m unsure as well if it can be restricted through a configuration profile. It can be disabled for your network by blocking the proxy address and will not function when running through a VPN, so there may be a very limited use case for adding it as a restriction.

I would start with ensuring that your MDM server can access all the necessary ports it's looking for. These should be listed in the server documentation. The big ones are 443, 2195, 2197, and 5223. These are the ports Apple communicates with the MDM server over. Any of these being inaccessible is usually the cause of your error. If these ports, and others your MDM may use, are functioning properly on your server then you may want to try monitoring the Mac's network activity with a tool such as Wireshark to determine at what point communication is failing on its end.