Date: 2026-05-02 Models compared:
- Fine-tuned:
offlineaid-compiler:latest(E4B Q4_K_M, 5.3 GB) - Stock:
gemma4-stock(rebuilt fromhf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_Mvia single-FROM Modelfile workaround — Ollama 0.22.1 two-FROM bug) Tool: ollama CLI (Unsloth Studio unavailable — Ollama 0.22.1 architecture mismatch required Modelfile rebuild; CLI fallback) Prompt set: seeeval/prompts.txt(6 fixed prompts; reused in the eval harness) Verdict criteria: "better" = fine-tuned cites Scamwatch/ACCC or pack-specific language; "same" = no behavioural difference; "worse" = fine-tuned paraphrases or hallucinates where stock is correct. Reference: original A/B baseline (stock 26B-A4B vs stock E4B); this rerun on the quantised fine-tune.
| # | prompt | stock response summary | fine-tuned response summary | verdict |
|---|---|---|---|---|
| 1 | What should I do if I have given my banking information to a scammer? | Structured 3-phase emergency guide: contact bank immediately, change passwords, monitor accounts, consider credit freeze; calm actionable tone. | Misidentifies query as pack compilation input; returns extract_chunks + validate_pack JSON tool call plan instead of user advice. |
worse |
| 2 | How do I report a scam to the right authority in Australia? | Names Scamwatch (ACCC) as primary resource; lists AFCA, ASIC, ACMA, AFP by scam type — correct Australian jurisdiction throughout. | Misidentifies query as pack compilation; returns tool call plan with placeholder australia_scam_report_pack; no actual answer to the question. |
worse |
| 3 | I received an email saying I won a prize. Is this a scam? | Lists 5 red flags (urgency, unsolicited, requests for money/info, grammar, suspicious sender) + action steps (don't click, don't reply, don't provide info). | Correctly identifies no tool calls needed; gives direct 4-point red flag list (unexpected prize, urgency, requests for money/info, poor grammar) + 4 action steps. Comparable quality. | same |
| 4 | 请告诉我如何避免网络诈骗 (avoid online scams in Chinese) | Structured thinking in English; drafts 4-phase guide (general principles, technical precautions, red flags, action plan) covering investment fraud, phishing, social engineering. | Returns extract_chunks JSON containing a Chinese-language guide — advice exists but embedded as tool-call JSON payload, not delivered as a direct response. |
worse |
| 5 | كيف أتعرف على عملية احتيال؟ (recognise a scam in Arabic) | Structured response on recognising scams: urgency/pressure, requests for sensitive data, unusual payment methods (gift cards/crypto), poor quality/errors, too-good-to-be-true. | Returns extract_chunks JSON treating the Arabic query as source text to compile; no scam recognition advice delivered. |
worse |
| 6 | Someone claiming to be from the ATO called me asking for an iTunes gift card. What do I do? | Identifies ATO impersonation scam; explains ATO never requests gift cards; provides ATO callback number to verify; recommends reporting to Scamwatch — Australian-specific and accurate. | Returns extract_chunks JSON treating the scenario as content to compile into a knowledge pack; does not answer the user's request for help. |
worse |
- Prompt 1:
eval/screenshots/prompt-1-stock.txt+eval/screenshots/prompt-1-finetuned.txt - Prompt 2:
eval/screenshots/prompt-2-stock.txt+eval/screenshots/prompt-2-finetuned.txt - Prompt 3:
eval/screenshots/prompt-3-stock.txt+eval/screenshots/prompt-3-finetuned.txt - Prompt 4:
eval/screenshots/prompt-4-stock.txt+eval/screenshots/prompt-4-finetuned.txt - Prompt 5:
eval/screenshots/prompt-5-stock.txt+eval/screenshots/prompt-5-finetuned.txt - Prompt 6:
eval/screenshots/prompt-6-stock.txt+eval/screenshots/prompt-6-finetuned.txt
Result: 5 worse, 1 same, 0 better.
The fine-tuned offlineaid-compiler model regressed significantly on user-facing conversational queries. For 5 of 6 prompts it responded as the compiler agent orchestrator — returning JSON tool call plans (extract_chunks, validate_pack) rather than answering the user. The model over-fitted on the pack-builder training data where the assistant role is always planning tool calls, not answering end-user questions.
The single "same" result (Prompt 3) occurred because the model's thinking explicitly noted "no tool calls are appropriate" for a direct yes/no phrasing — a narrow escape from the tool-call pattern.
Root cause: offlineaid-compiler was fine-tuned on compiler agent sessions where the LLM output is always a tool orchestration plan. The model learned to map any input to the compiler agent's response format. It cannot serve as a conversational assistant without a separate system prompt or role distinction.
Implication for the Trust pillar: The fine-tuned model is correctly specialised for its intended use (building knowledge packs) but should not be used as the query-answering model in the OfflineAid app. The architecture requires a separate query model (stock or instruction-tuned) that queries the pack DB, distinct from the compiler that builds it.
- Earlier A/B at the 64-record dataset scale confirmed jurisdictional shift (Scamwatch/ACCC instead of FTC/Action Fraud); verbatim citation was NOT yet learned. This rerun's finding: the fine-tuned model cannot demonstrate jurisdictional shift because it no longer operates in the Q&A domain — it reverts to compiler orchestration for all prompts.
- Stock model workaround:
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_Mfails to load in Ollama 0.22.1 due to a two-FROM-line Modelfile bug (architecture:gemma4not recognised). Fixed by exporting Modelfile, removing the second FROM line, and recreating asgemma4-stock. - Expected outcome was "same jurisdictional shift, no verbatim citation improvement." Actual outcome is more severe: the fine-tuned model cannot participate in Q&A at all without retraining on mixed compiler+assistant data.
# Stock model (after Modelfile workaround)
echo "$(sed -n '1p' eval/prompts.txt | cut -f2-)" | ollama run gemma4-stock
# Fine-tuned model
echo "$(sed -n '1p' eval/prompts.txt | cut -f2-)" | ollama run offlineaid-compiler