Additional info: I noticed that whenever this happens, there are two AXUIElementDestroyed notifications posted.

It seems this can happen if an alert is opened as a result of a menu selection, or if a new alert is opened immediately after closing an old one, and it seems to go away if I add a half-second delay before opening the new alert. I might make a bug report later.