For others reading this thread, it was IPv6 connections to timestamp.apple.com were unreliable. If you don't need IPv6, then you can set it to "link-local only", which will force codesign to use IPv4.
Thanks to @eskimo for the assist.
Topic:
Code Signing
SubTopic:
General
Tags: