Or any document that indicates this not feasible.

I used to have this problem too. But later I figured out that cause the resolution of YUV and depth image are not the same. For example, your YUV image is 4 times wider and 4 times higher, so when you using the cameraIntrinsics from YUV on depth image, which is like 200 * 250, you are supposed to divide the intrinsic parameters by 4. eg, focalx = 1464.9269 / 4, focaly = 1464.9269 / 4, cx = 960.94916 / 4, and cy = 686.3547 / 4

exactly the same