I close this thread myself. What I observed is not correct.

I noticed that on https://developer.apple.com/documentation/networkextension/packet_tunnel_provider Note When a VPN configuration is active, connections use the VPN instead of iCloud Private Relay. Network Extension providers also don’t use iCloud Private Relay. and this post https://developer.apple.com/forums/thread/682274. it seems that the Private Relay should have the lowest precedence, but my test shows that the traffic from the browser still being routed to the Private Relay.

Thanks Matt. I just want to add some more observe results: On an iPhone 5s iOS 12.5.4 run the same application the Message too long doesn't happen no matter it's on WiFi or Cellular network; On an iPhone 7 iOS 14.6, it's very easy to reproduce this issue on a WiFi network, but I never see it happen on a Cellular network; I didn't change the MTU in the code but set the tunnelOverheadBytes instead. When it's set to be 44 or 0 the same issue happens; If ignoring the EMSGSIZE error when it happens the setReadHandler seems to run into a Message too long error dead loop, so when it happens I have to restart the UDP session.

I also got this error Error Domain=NSPOSIXErrorDomain Code=40 "Message too long" with NWUDPSession setReadHandler method on iOS 14.6, WiFi network.