Yes, they are very similar, but that's by design.
What I'm trying to accomplish is to have a series of images which are identical on one side and different on the other side. I want them to match each other. So I took a series of photos. They all have the same stuffed animal on the left side, and different ones on the right side. Except for one image, which has no stuffed animal on the right side. The idea is that the image with one animal should match any of the ones with two. So the one with one animal is tracked, and the others are scanned. This actually works fine.
I was testing and reversed which image was tracked and which was scanned, and that's when I noticed this behavior. It doesn't happen if both images have two stuffed animals.. they match in both directions. It only happens with the image that has only one.
It won't interfere with what I'm trying to do, but the behavior didn't make any sense to me so I wanted to understand what's going on and if it's a sign that I'm doing something wrong.