Change Sign In With Apple button style based on current color scheme

I'm having some issues with SwiftUI's SignInWithAppleButton's signInWithAppleButtonStyle. I am trying to change the color of the button based on the user's current scheme or if it changes. This is iOS 14 and SwiftUI:

Code Block 
@Environment(\.colorScheme) var currentScheme
@State var appleButtonWhite = false
VStack{
SignInWithAppleButton(
                .signUp,
                onRequest: { request in              
                    request.requestedScopes = [.fullName, .email]
                },
                onCompletion: { result in
                    switch result {
                    case .success (let authResults):
                        print("Authorization successful.")
       
                    case .failure (let error):
                        print("Authorization failed: " + error.localizedDescription)
                    }
                }
            )
            .frame(minWidth: 0, maxWidth: .infinity)
            .signInWithAppleButtonStyle(appleButtonWhite ? .white : .black)
}
.onChange(of: currentScheme, perform: { (scheme) in
        if scheme == .light
        {
            appleButtonWhite = false
        }
        else
        {
            appleButtonWhite = true
        }
    })

When appleButtonWhite changes values, it renders the view as it should since the state is changing. When I debug the button it has the correct appleButtonWhite value, but for some reason the style just never changes. I have no idea why. I have done many style changes in my code with regular buttons and it works correctly based on different states. Any ideas why Apple's doesn't change?