If you implement the method -(BOOL)worksWhenModal to return YES on the target object for your menu item, AppKit will allow the menu item to be validated, and enabled if the target implements the item's action.
Still doesn't seem to work for the sample code I provided. I had to leave the menu item's target to nil to make it work, but then using a normal window was enough, without the need to implement worksWhenModal.
This is in fact the workaround I just found. If I leave the menu item's target to nil and make sure that the view of the view controller or a subview is first responder, e.g. with
override func viewDidAppear() {
view.window?.makeFirstResponder(view)
}
then the menu item is enabled. In my case it doesn't work though, since the view controller is a child of another view controller and it's not guaranteed that the view of the view controller showing the menu (and implementing the menu item actions) is first responder (as the parent can have other views that can be first responders). As soon as I set the menu item's target to self, the menu items are disabled again.