After many hours of digging, it turned out that another possible bug caused this weird problem.
It seems AppDelegate applicationShouldHandleReopen is always called no matter if there is an existing running instance. This is true no matter if a project is created using XIB based style or new storyboard style; it's also not related to languages (OBJC or Swift). I verified this on newly created projects.
My app has some custom actions to do in applicationShouldHandleReopen that eventually shows the main window. Since now it's called unconditionally, it will show main window on launch.
I also verified on my old MBP, applicationShouldHandleReopen is NOT called on app launching during debugging.
Is this applicationShouldHandleReopen behavior by design on new macOS versions (13+)?