Had an epiphany that turned out correct. The problem is I was accessing the texture I was writing into from a separate thread. My guess is on certain frames it was accessing it in the middle of a render pass. Solved the issue by creating another texture, blitting the result of my command buffer upon completion and passing it back to the other thread.