Figured out the issue, turns out I needed to run this using Python 3.11.12. Version 3.12 and 3.13 would cause this missing module error. In the tutorial it says Python 3.11+ is required but clearly it has to be just some variant of 3.11. Also it only worked on my Mac, not on any other Linux system.

Tried this out again with v2 of the adapter training toolkit. Still the inference uses 26GB!!! And 40GB was used after one epoch. Really hope either there's improvements made to the toolkit or more clarity around the hardware specs needed for tuning the model

@NavaneethanGanesan Still not an acceptable answer. Reducing the number of tokens outputted to alleviate the memory issue is the equivalent of saying the best way to get rid of the bugs is to delete the code. Sure I can reduce the max tokens to a measly 64 but how does this help me test the models and what I'm building if it requires more than 64 tokens (which is common for many LLM applications nowadays)

@Opsroller While that may be true, this is a 7B parameter model. 512GB is overkill for a 7B model. To fine-tune a 7B model with LORA, you should be able to get this done with 64GB RAM at most. Even on the VM I spun up, there were instances where my VM ran out of memory. I believe there's some issue with the sample code provided by Apple and potentially a memory leak that's causing excessive memory usage.