Xcode Server exists as a part of the Xcode app, so you can't remove it... but you can reset to a state where it is not configured.
Swift 'Off' Xcode Server in the Xcode preferences.
In terminal, run sudo xcrun xcscontrol --reset
Remove the Xcode Server account in the Users system preferences.
Topic:
Developer Tools & Services
SubTopic:
Xcode
Tags: