I missed your suggestion to use UnlitSurface, I didn't think the lighting would affect texture sampler readings, but it does seem to for some reason. That went away with unlit surfaces though. Thanks for the help!