I was facing a similar problem. You will have to resolve the rejection issues first. Then only the portal will let you remove the app.