Alarms scheduled using AlarmKit is not reliable

The alarms that I am scheduling using alarmKit is not reliable at all, especially in beta 8 (Used to have this issue in previous betas, But they were inconsistent).

The alarm fires anywhere after 5 mins to 45 mins from the actual scheduled time.

I'll share the relevant code block here in case if it is me who is doing something wrong.

This code block doesn't throw any errors and I am also able to see the scheduled alarms when I query them. The issue is with alarms not firing on the specified time

let alert = AlarmPresentation.Alert(
    title: "Wakey Wakey!",
    stopButton: .init(text: "stop", textColor: .red, systemImageName: "stop.fill")
)

let presentation = AlarmPresentation(alert: alert)

let attributes = AlarmAttributes<CountDownAttribute>(
    presentation: presentation,
    metadata: .init(),
    tintColor: .orange
)

let schedule = Alarm.Schedule.fixed(alarmTime)

let config = AlarmManager.AlarmConfiguration(
    schedule: schedule,
    attributes: attributes
)

let id = UUID()

do {
    let alarm = try await AlarmManager.shared.schedule(id: id, configuration: config)
    print("Alarm set successfully for \(alarmTime) - \(alarm)")
} catch {
    print("Failed to schedule alarm: \(error)")
}