NEPacketTunnelProvider died unexpectedly, no crash dump

App & System Services Networking
mike.ly OP
Created Aug ’17
Replies 20
Boosts 0
Views 11k
Participants 12

Hello,

I'm trying to debug an intermittent crash in my NEPacketTunnelProvider. Basically, it will sometimes crash about a minute after establishing a connection. If it survives the first couple minutes, it will work fine for the duration of the connection.


My main problem is that there doesn't appear to be any crash dump created. My provider just stops logging, and I see the following logged in the Console:


neagent: extension connection was interrupted
neagent: Extension <my extension ID> died unexpectedly
nesessionmanager: NESMVPNSession[<uuid>]: status changed to disconnecting


The crash happens only on macOS; iOS is fine.


A few things I've tried:

  • Checked for a crash dump in ~/Library/Logs/DiagnosticReports and /Library/Logs/DiagnosticReports; found nothing related to my provider.
  • Attached the Xcode debugger. I can never get it to crash with the debugger attached (of course 😝).
  • Monitored memory usage in Xcode. It's typically around 6.7MB, but can drift up close to 8MB during heavy traffic. I believe this is acceptable?
  • Added an uncaught exception handler via NSSetUncaughtExceptionHandler. But my handler never gets called; I suspect this isn't supported for extensions?
  • Followed instructions for Crash Logs and VPN Logging.


Has anybody run into an issue like this? Is there some other way to gather info that I'm missing?


Thanks,

Mike

Replies  20
Boosts  0
Views  11k
Participants  12
DTS Engineer OP
Apple
Aug ’17

The crash happens only on macOS; iOS is fine.

This sounds very much like a problem I’ve seen reported by other developers (r. 32073323). On macOS only, if you stop and then immediately (within 5 seconds) start a packet tunnel provider, the provider will spontaneously stop about 20 seconds later. The problem is still being investigated by R&D engineering.

Does that match what you’re seeing?

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
mike.ly OP
Aug ’17

Yep, that sounds like it. Is there any info I can provide that would help the investigation?

DTS Engineer OP
Apple
Aug ’17

Is there any info I can provide that would help the investigation?

No, that’s not necessary. I can reproduce it with my own tiny packet tunnel provider test project.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
mike.ly OP
Sep ’17

FYI I can still reproduce this bug on macOS 10.13 Beta 9.


Are you able to share any progress/updates regarding a fix? Thanks!

DTS Engineer OP
Apple
Sep ’17

Are you able to share any progress/updates regarding a fix?

I’ve nothing to concrete to report, alas.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
ericm OP
Oct ’17

We're seeing it consistently also, on 10.12 and 10.13. Exact same repro as Quinn described.
mike.ly OP
Oct ’17

This is still an issue in macOS High Sierra 10.13.1 beta 2 (17B35a).
xfx.dev OP
Apr ’18

Still crash on macOS High Sierra 10.13.4(17E199)
roee84 OP
Dec ’18

Seems like it's still crashes on macOS Mojave 10.14.1
zx2c4 OP
Feb ’19

The solution to apple bug 32073323 is to add a call to `exit(0)` to the provider's stopTunnel method. For example:


override func stopTunnel(with reason: NEProviderStopReason, completionHandler: @escaping () -> Void) {

// Do your normal tunnel stopping stuff


completionHandler()


#if os(macOS)

// HACK: This is a filthy hack to work around Apple bug 32073323. Remove it when

// they finally fix this upstream and the fix has been rolled out widely enough.

exit(0)

#endif

}
keeshux OP
Jun ’19

It's been 2 years already. This is still around in Mojave and is crazy stuff.


It's unacceptable for such a severe bug to live this long without any feedback from Apple. No surprise as I have had other major issues and they pretty much received the same treatment, despite being properly and thoroughly reported. Ignored, marked as duplicate (i.e. no updates), stuck forever.


I kind of hate this behavior, because in this case I have a decent user base for my app and people legitimately need answers. Everybody in software is asked deadlines. I can't possibly believe that in 2 years nobody looked into what is a severe bug nonetheless. More so given that the bug can be reproduced consistently.


Absurd.
keeshux OP
Jun ’19

I'll try this ASAP. Did you find it by luck or through advice by an Apple agent?
jake144 OP
Aug ’19

any luck with this solution?

DTS Engineer OP
Apple
Aug ’19

With regards this bug (r. 32073323), I’m sad to report that there’s still no fix in sight here )-: However, the workaround suggested by zx2c4 does seem to help.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
jaroslavqwerty OP
May ’20

Anything new after one more year passed?

I think I have the same problem, but using systems IKEv2. If mac goes to sleep for about 5 minutes, after it wakes up, VPN connection gets dropped ant it reconnects. Then after 20 seconds that new connection drops.

In console I can see this log: `Extension com.apple.NetworkExtension.IKEv2Provider died unexpectedly`.


Looks like it got as lot worse in Catalina, especially the latest 10.15.4.


If you need any information I'll be glad to provide it.

DTS Engineer OP
Apple
May ’20

If you need any information I'll be glad to provide it.

This is the first time I’ve seen this reported against the built-in VPN transports, although that doesn’t surprise me given the common underlying infrastructure. However, it is a new wrinkle and that definitely makes it worthwhile for you to file a new bug about it.

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
jaroslavqwerty OP
May ’20

OK, I've filled a report: FB7716038.

Hope this will be fixed soon, because all the VPN clients that are using native IKEv2 implementation suffer from this issue.
auryisaghost OP
Jul ’20
I also encountered this bug while creating a transparent proxy (NEAppProxyProvider in extension, NETransparentProxyManager in host app). Although the solution provided by zx2c4 works to stop the proxy from crashing 15-20 seconds after it starts, it leads to another problem: when the proxy is disconnected gracefully (e.g. user clicks "Disconnect" in System Preferences), the proxy will hang on "Disconnecting" for about 5 seconds before finally exiting.

I don't know what exactly causes this "Disconnecting" problem, but my guess is that calling exit(0) immediately after completionHandler() in stopProxy does not give sufficient time for the proxy to communicate its successful close to the remote network service manager before the proxy exits. Therefore, the network service manager just waits a certain amount of time before assuming the proxy crashed and moving it to the "Not Connected" state.

My solution is to call exit(0) as suggested by zx2c4, but immediately before that, sleep for half a second with Darwin.usleep(500000). Of course, that adds a ~0.5 second delay on exit, but it's much better than ~5 seconds. I'm aware this is even more filthy than the already self-described "filthy hack" by zx2c4, but... it seems to work.

So the final code should look something like:

Code Block 
override func stopProxy(with reason: NEProviderStopReason, completionHandler: @escaping () -> Void) {
	/* ... shutdown work... */
	completionHandler()
	#if os(macOS)
	/* Workaround for apple bug 32073323. */
	Darwin.usleep(500000)
	exit(0)
	#endif
}

DTS Engineer OP
Apple
Jan ’22

This workaround is also needed for Network System Extension?

I haven’t tested it but, based on my understanding of how NE uses the underlying system support for app and system extensions, I suspect that the workaround isn’t necessary for an NE sysex. However, that’s just a guess. You should suck it and see.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
callmepali OP
Apr ’24

Once in a while we get back to this thread and I would like to report that we are not able to reproduce this issue anymore (will update this post if things change).

  • We use NEPacketTunnelProvider
  • All our testing was done on macOS 14

Anyone else experienced improvements or are we just lucky in our testing so far?

NEPacketTunnelProvider died unexpectedly, no crash dump
First post date Last post date
 
 
Q