Sorry, missed the part about placement of the call

It seems that tf.config.set_visible_devices([], "GPU") works in tensorflow but not with tensorflow-metal RuntimeError: Visible devices cannot be modified after being initialized