Learning experiment
Dolphin QLoRA: why lower loss was not enough
Two local Dolphin 8B fine-tuning runs: a failed adapter, a lower learning rate, validation-based selection, and measured classification results.
The first fine-tuning run made the model worse even while validation token loss fell. The follow-up turned that failure into a study of generated-answer evaluation and checkpoint selection on Apple Silicon.
A small, measurable classification task
The task is to classify customer-support messages into 27 intents and return a label in JSON. Dolphin3.0 Llama3.1 8B is converted to a 4-bit MLX base, while QLoRA trains a small separate adapter with the base weights frozen.
The documented full runs use 1,080 training examples. A fixed suite of 297 classification questions provides a concrete comparison between the base model and the two adapters.
Failure, diagnosis, and a better follow-up
The base answered 210 of 297 questions correctly, or 70.71%. The first adapter scored 54.21%. Falling token loss had not guaranteed that the model would generate the right labels.
A fresh run used a learning rate of 0.00001 instead of 0.0001 and checked generated validation answers every 50 updates. The checkpoint selected at step 450 scored 264 of 297, or 88.89%, on the same final classification suite.
What the improvement actually shows
The follow-up changed both the learning rate and checkpoint-selection method, so it does not isolate their individual effects. One seed and a public synthetic dataset also limit generalization.
The final questions had already been evaluated during the first run. The result is a comparison on this reused suite, not an untouched benchmark or a production accuracy promise. Three coherent general-answer probes are only a small qualitative check.
Follow the evidence rather than the headline
The repository preserves predictions, settings, diagnostics, and checkpoint-selection records. The experiment and reproduction guides explain what changed and how to inspect the results without treating a falling loss curve as sufficient evidence.
Sources and further reading
Based on the linked project documentation, reviewed on 2 November 2026. Capabilities and limitations describe that documented version.
View repository