I ended up getting a Colab subscription and ran on an A100. L4 still OOM on full precision.

Using precision="bf16" makes the model small enough while using swap but also makes the model unusable... > display(Markdown(output[0].response)) Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Vac Do other people get this too?

I'm also trying to generate adaptors the 3B model and running out of memory on a 16GB Mac Mini. I'll have more time today to dig deeper but guidance would be appreciated. I found this writeup someone did https://collisions.substack.com/p/fine-tuning-apples-new-foundation?utm_campaign=post&utm_medium=web&triedRedirect=true but they ended up renting a H100. Edit: base-model.pt is already 12.7GB