Fear not.
Homebrew will uninstall cleanly, as you have found.
Xcode is supplied by Apple... I think we can assume that Apple are a trusted source?
You can uninstall Xcode by dragging it's icon (from Applications) to the Trash.
If you have used Xcode (before uninstalling it), then it may have created some folders, elsewhere on your drive...
...but unless you are pretty obsessive about these things, they can be safely ignored.
Alternatively, a quick web search should uncover a full list of any extra Xcode files and folders, that you can safely delete, if you want to.
Topic:
Developer Tools & Services
SubTopic:
Xcode
Tags: