Looking at your first image, the traditional GPU solution would be to use the stencil buffer to only paint each pixel once.
If you have anti-aliased edges, the binary stencil buffer isn't perfect. But you can use the alpha channel in the framebuffer to get a similar but fractional effect.
Or you can re-triangulate your triangle strips on the CPU to get a geometry that doesn't self-intersect. You should aim to do that ahead-of-time, but it's not completely impossible to do it for every frame.
In your other images with the brush texture, I think that again using the framebuffer alpha as a pseudo-stencil is probably the best solution.
Topic:
Graphics & Games
SubTopic:
Metal
Tags: