Generally it's best practice to not give keys to developers, but to your managed infrastructure. In other words put the key on CI, and limit access to individuals if you can. If a developer who had direct access to keys left, it's also a good idea to revoke that certificate yes. Often this just means distributing the new key/cert to your infra.