Ok, I have found the answer. It was because of the iOS version. I was using iOS 15.1 previously. After upgrading all my devices to iOS Beta 15.2, they worked perfectly. I even didn't rebuild the app, all I did was upgrading the iOS version. The interesting thing is that, when I was mixing these two iOS versions together (say device A was iOS 15.1 and device B was iOS 15.2), I could see device B's ARParticipantAnchor in device A's screen but I could not see device A's ARParticipantAnchor in device B's screen. The problem seems to be that ARKit under iOS 15.1 could not output ARCollaborationData with proper ARParticpantAnchor transform information, so the receiver device could not interpret that data correctly.

I upgraded Xcode to 13.2.1 and problem solved.

I got the same problem on Xcode 13.3