Before and after numbers from a base-vs-fine-tuned comparison on a small classification task. This page shows you exactly how we report fine-tuning results: the task, the methodology, and the table the eval harness produces.
Classifying incoming support tickets into 6 categories — Billing, Account access, Bug report, Feature request, How-to, Cancellation — on a small, public-style dataset of labeled support tickets. We picked it because it is the shape of a real business problem (routing tickets to the right queue) without exposing anyone's data.
| Step | What we do |
|---|---|
| Dataset | 1,500 labeled support tickets, 6 categories, de-duplicated and reviewed. |
| Train / eval split | 1,200 tickets for training, 300 held out for evaluation. The eval set is frozen before training starts and never touched during tuning. |
| Metric | Accuracy plus macro-F1 (so a model can't hide poor performance on a rare category behind a common one). Per-category scores reported. |
| Eval examples | 300 — the same frozen set is scored for both the base model and the fine-tuned model. |
| Harness | The eval harness in the llm-fine-tuning-toolkit repo (being prepared for public release) runs both models over the eval set and computes the table below. |
Illustrative example values — run the harness on your own data for measured results. Structure of the table the harness produces:
| Model | Accuracy (300 eval examples) | Macro-F1 | Notes |
|---|---|---|---|
| Base model | 78.3% (example) | 0.76 (example) | Zero-shot with the frozen classification prompt. |
| Fine-tuned | 91.7% (example) | 0.90 (example) | Trained on the 1,200-ticket train split, scored on the same frozen 300. |
The numbers above are placeholders showing the table's shape — a real run replaces them with computed values and adds per-category rows. We never present fine-tuning results without the methodology block above; a number without its split, metric, and eval size is marketing, not measurement.
No. The task framing and the numbers in the table are an illustrative example of how an eval readout looks. The real, measured numbers come from running the harness in the llm-fine-tuning-toolkit repo on your own labeled data.
Accuracy and macro-F1 on a held-out eval set the model never trained on, plus per-category scores so you can see exactly where the fine-tuned model improved and where it did not.
Our internal repo (being prepared for public release) that standardizes the fine-tuning loop: dataset prep, train/eval splits, the training run, and the eval harness that produces the comparison table. It is referenced here by name only — it is not published yet.
When you have hundreds of labeled examples, the task is stable, and prompt engineering has plateaued. If you have twenty examples and a shifting task, fine-tuning is premature — the free assessment will tell you which camp you are in.
VROOM ANALYTICS