Was this ever clarified by an Apple employee? It doesn't seem like we should have to tick the diagnostics box based on the statement: "You are not responsible for disclosing data collected by Apple." for crash logs and diagnostics provided in the xcode interface, otherwise every app in the app store would have to have this option checked if it was required.

The WeatherKit option is un the Capabilities tab not App Services. You also need to register a WeatherKit key in the top level Keys section in the developer site. Once you have completed both of those step, this question has information on how to call the REST API using a JWT: https://developer.apple.com/forums/thread/707418

Did you enable WeatherKit in the apps Signing & Capabilities section for the app?

Unfortunately, entering my real apple ID/password does not work. It show success but the upgrade/downgrade never occurs the UI still shows the original subscription level. Signing out of the production AppleID and only having the sandbox tester on the device doesn't work either. In that scenario the password box is displayed but with an empty username in the dialog, entering the sandbox password just fails. I'm confused how people are successfully testing these scenarios :/