You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: plug_in_modules/qlora_adapter.py
+6Lines changed: 6 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,9 @@
1
+
# The bitsandbytes library has historically been Linux/CUDA only — Mac MPS support is improving, but not guaranteed on the M-series chip.
2
+
# On a Mac Mini, test with pip install bitsandbytes and check if 4-bit quantization works on MPS.
3
+
# If not, the fallback is 8-bit or straight float16 without quantization, which will use more RAM but will run.
4
+
# The Ollama bridge at the bottom (export_to_ollama_modelfile, merge_adapter_weights) is the path to getting a fine-tuned adapter running locally — you train with HuggingFace, merge the LoRA weights back into the base model, then import to Ollama.
5
+
# The CAIOS.txt system prompt gets baked into the Modelfile, so it loads automatically without needing to pass it every call.
0 commit comments