Selecting ~/Library in open panel doesn't give access to ~/Library/Mail

A user of my app brought to my attention that unless they select their ~/Library/Mail folder explicitly in an open panel, they get an error when scanning it inside my app. I can confirm that I also get a permission error when trying to scan it as a subfolder of ~/Library, but not if I select it directly.

I'm assuming this is intentional, but it would be nice to have an explanation or some documentation that I can point my users to when they encounter what appears to them as a bug in my app. What makes this matter even more confusing is that selecting a folder in any open panel of an app gives the app access to it for the lifetime of the app, but after restarting the app, access is lost again (unless it has a bookmark to it). This was probably the reason why the user thought that it worked in another app but not in mine.

This is the code I use to scan:

let openPanel = NSOpenPanel()
openPanel.canChooseDirectories = true
if openPanel.runModal() == .cancel {
    return
}
let enumerator = FileManager.default.enumerator(at: openPanel.urls[0], includingPropertiesForKeys: nil) { url, error in
    print(url.path, error)
    return true
}
while let url = enumerator?.nextObject() as? URL {
    
}

And this the error related to the Mail folder:

~/Library/Mail Error Domain=NSCocoaErrorDomain Code=257 "The file “Mail” couldn’t be opened because you don’t have permission to view it." UserInfo={NSURL=file:///~/Library/Mail, NSFilePath=/~/Library/Mail, NSUnderlyingError=0x600002991470 {Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"}}