For anyone who ends up here after a search: Turns out you can't delete the version and there's no rename button, but the version number is an editable text field that you can just change and Save.
Topic:
App Store Distribution & Marketing
SubTopic:
App Store Connect
Tags: