Some more experimentation, this doesn't happen with the pen tool as far as I can tell, only the pencil...