Tinman Programme
Training factory · run report
Sixteen training jobs, five days, one shipped model
A complete record of the Tinman fine-tuning pipeline as it actually ran between 26 and 30 July 2026 — every job, every loss curve, and what each number does and does not tell you. Two model families were trained: Qwen3-8B, which became the shipped model, and Gemma-4-E4B, which was evaluated and set aside.
The window at a glance
Everything in this report measures throughput and training dynamics. None of it measures whether the model gives correct legal answers. Those are different questions, and a factory report is exactly where they get confused. The loss curves below show that the training took; they are silent on whether what was learned is right.
How the pipeline ran
Corpus assembly
South African labour-law sources — statutes, CCMA material, and case law — chunked into instruction/response pairs with citation markers, then filtered for boilerplate, template spray, and missing abstention behaviour.
Serverless SFT on SageMaker
Supervised fine-tuning with LoRA adapters (rank 32, alpha 64, dropout 0.05) against the
SageMaker public-hub base models. Fourteen of the sixteen jobs are serverless, billed per
token, which is why the ledger reports tokens rather than instance-hours. The two earliest
attempts ran on managed ml.g5.xlarge instances instead, failed within eight
minutes, and were abandoned in favour of the serverless path.
Sweeps and probes
A learning-rate sweep, an 8K sequence-length probe, and a stock baseline were run as cheap controls. Three further runs were stopped by hand once their curves showed the configuration was wrong — visible as outlined bars in panels 05 and 06.
reset-v3 and registration
The final Qwen3-8B run: 336 optimiser steps over 2 epochs at 4096 tokens, cosine schedule with 10% warmup, merged weights, registered to the model package group as version 8.
Training dynamics
Job ledger
Every training job in the window, oldest first. Token counts are what AWS metered; a dash means the job failed before metering began.
| Started | Job | Family | Status | Steps | Minutes | Billable tokens |
|---|---|---|---|---|---|---|
| 2026-07-26 06:26:49 | Tinman-Legal-T1 | Qwen3-8B | Completed | 1 | 98.6 | 30,451,417 |
| 2026-07-26 07:54:16 | qwen3-8b-stage1-corrected | Qwen3-8B | Failed | — | 7.8 | — |
| 2026-07-26 08:09:26 | qwen3-8b-stage1-corrected | Qwen3-8B | Failed | — | 5.5 | — |
| 2026-07-26 08:27:34 | qwen3-8b-stage1-fixed-seqlen | Qwen3-8B | Completed | 1 | 96.7 | 30,451,417 |
| 2026-07-26 10:09:25 | gemma4-e4b-stage1 | Gemma-4-E4B | Completed | 29 | 36.5 | 38,541,173 |
| 2026-07-26 12:40:57 | qwen3-8b-stage2 | Qwen3-8B | Completed | 75 | 35.0 | 8,685,643 |
| 2026-07-26 13:50:49 | qwen3-8b-stage3-corrective | Qwen3-8B | Failed | — | 6.0 | — |
| 2026-07-26 14:05:44 | qwen3-8b-stage3-corrective | Qwen3-8B | Completed | 8 | 9.3 | 60,038 |
| 2026-07-29 09:36:01 | qwen3-8b-lrsweep-1e4 | Qwen3-8B | Completed | 166 | 33.7 | 9,728,848 |
| 2026-07-29 10:37:17 | qwen3-8b-reset-v2-e8 | Qwen3-8B | Stopped | 166 | 31.0 | 9,728,848 |
| 2026-07-29 11:09:54 | qwen3-8b-reset-v2-e3 | Qwen3-8B | Stopped | — | 0.8 | — |
| 2026-07-29 11:14:59 | qwen3-8b-probe-8k | Qwen3-8B | Completed | 166 | 33.1 | 9,728,848 |
| 2026-07-29 11:53:34 | qwen3-8b-reset-v2-e3 | Qwen3-8B | Stopped | 132 | 54.4 | 17,107,457 |
| 2026-07-29 12:50:50 | qwen3-8b-reset-v2-e3c | Qwen3-8B | Completed | 166 | 89.6 | 29,184,635 |
| 2026-07-29 14:24:14 | qwen3-8b-stock-baseline | Qwen3-8B | Completed | 166 | 33.7 | 9,728,988 |
| 2026-07-30 00:15:48 | qwen3-8b-reset-v3 | Qwen3-8B | Completed | 168 | 59.2 | 19,411,430 |
What it cost
Cost Explorer reports $151.53 of Amazon SageMaker usage across 24 July – 1 August 2026, offset by $151.53 in promotional credits, for a net cash outlay of $0.00. Both numbers are worth stating. The credit is a one-time position; the list-price figure is what this pipeline costs to run again, and it is the number to plan against.
Serverless training bills per token, not per instance-hour, so the four largest jobs by cost are the four largest bars in panel 05 — not the longest-running ones. The three hand-stopped runs still consumed roughly 28 million billable tokens between them, which is the real price of the sweeps.
Caveats that change the reading
The two loss curves are not directly comparable
Qwen and Gemma were trained through different trainer stacks and log different metric
names — gpt_loss against train/loss. They also use different
tokenizers, so "loss per token" is denominated in different tokens. More importantly, the
Gemma run was configured with thinking disabled while the Qwen run trains reasoning on.
Read each curve's shape against itself; do not read the gap between them as a
quality difference.
Gemma's run ended mid-schedule
Panel 04 shows Gemma's learning rate rising monotonically to 5e-5 and stopping there, never entering decay. Its final loss of 0.638 is therefore not a converged figure — it is where an unfinished schedule happened to stop. It is not evidence that Gemma is the weaker base model.
Validation loss for Qwen rests on two points
The Qwen trainer wrote two evaluations for the entire run (0.4161, then 0.4071). That is enough to say the held-out loss did not blow up, and not enough to characterise a curve. Gemma's twelve points are a genuine trajectory; Qwen's two are an endpoint.
Gemma's logged perplexity is wrong, so it is not plotted
The run reports train/perplexity of 1.027 alongside a loss of 0.638. Those
are inconsistent — exp(0.638) is 1.89 — so the metric is mislabelled at source. It is
excluded here rather than shown with a caveat, because a plotted number invites belief.
No held-out quality evaluation is reported here
A 104-question evaluation exists but is void and not cited: none of its prompts carried the ChatML wrapper the models were trained with, so the fine-tune was tested out of distribution and in the wrong thinking mode. What that run did establish is structural — the fine-tuned model produced IRAC-formatted answers where the stock model produced none. That is evidence the training took, not evidence the answers are legally correct. A valid evaluation is outstanding work.
Throughput is not quality
212.8 million tokens and 10.5 hours of training are measures of a pipeline that works, repeatably, end to end. They say nothing about the corpus underneath. Known open defects — mislabelled legislation chunks, an entirely first-instance case-law set with no appellate coverage — are documented separately and are not visible in any chart on this page.