Fine-tuning Gemma 4 E4B with LoRA to bring FormAI's coaching on-device

FormAI is a Kotlin Multiplatform (Android/iOS) app that analyses a video of your golf swing, basketball shot, running form and so on, and gives you coaching feedback. Today that works by sending the video straight to Gemini’s cloud API. This post is about a prototype exploring the other end of that spectrum: can we distil that same coaching behaviour into a small model — Gemma 4 E4B (4.5B effective parameters) — fine-tuned with LoRA, small enough to eventually run entirely on-device?

Everything below happened as one end-to-end local pipeline: real seed videos, a real Gemini API as the “teacher”, a real LoRA training run against the real model, a real merge, and a real conversion down to a format Android’s on-device runtime can load. Code snippets throughout are lightly trimmed for the post but otherwise exactly what ran.

The pipeline

1. Send seed clips through the same Gemini prompts FormAI already uses, to get “teacher” coaching responses.
2. Filter out clips where Gemini couldn’t actually critique the form (bad framing) rather than gave real feedback.
3. Extract still frames from each clip (Gemma’s vision tower takes images, not video).
4. LoRA fine-tune Gemma 4 E4B on the (frames, prompt, teacher response) triples.
5. Merge the adapter into the base weights.
6. Quantize and convert to LiteRT-LM format for on-device inference.

Step 1: Gemini as the teacher

FormAI’s existing analysis service sends the video inline to Gemini along with an activity-specific prompt:

shared/src/commonMain/kotlin/dev/johnoreilly/formai/videoanalysis/VideoAnalysisService.kt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
AnalysisType.GOLF_SWING -> """
    You are a golf swing coach.
    Analyze this swing video and provide concise, actionable feedback on stance, grip,
    backswing, downswing, impact, and follow-through.
    """.trimIndent()

private suspend fun performGeminiCall(apiKey: String, mimeType: String, base64: String, prompt: String): String {
    val model = "gemini-3-flash-preview"
    val url = "https://generativelanguage.googleapis.com/v1beta/models/$model:generateContent?key=$apiKey"
    val request = GeminiRequest(
        contents = listOf(Content(role = "user", parts = listOf(
            Part(text = prompt),
            Part(inlineData = InlineData(mimeType = mimeType, data = base64))
        )))
    )
    val response = httpClient.post(url) {
        contentType(ContentType.Application.Json)
        setBody(request)
    }.body<GeminiResponse>()
    return response.candidates?.firstOrNull()?.content?.parts?.firstOrNull()?.text ?: "No response text returned."
}

To build a training set, we need this same call run in bulk over a folder of seed clips rather than one at a time from the app. That became a small standalone tool, dataset-bootstrap, that mirrors this exact request shape and walks a directory of clips (organised one subfolder per activity type), writing out {video_path, analysis_type, prompt, teacher_response} rows to a JSONL file — resumable, so a half-finished run never re-spends an API call on a clip it already labelled.

For seed data we used short, rights-cleared Pexels stock clips of golf swings rather than anyone’s personal footage.

A data-quality problem the hard way: hedging responses

The first clip we ran through — a tight close-up on just the ball/club impact zone — didn’t get a critique back. It got:

“I cannot provide a complete analysis of your stance, grip, or the full arc of your swing… please provide a face-on view and a down-the-line view.”

That’s a real Gemini response, and it’s a correct one — the clip genuinely doesn’t show enough. But training a model on rows like that teaches it to hedge and ask for better footage instead of actually coaching, which is the opposite of what we want. So the bootstrap tool grew a simple filter:

tools/dataset-bootstrap/src/main/kotlin/…/Main.kt

1
2
3
4
5
6
private val HEDGE_PHRASES = listOf(
    "can't see", "cannot see", "not visible", "re-record", "unable to",
    "i cannot provide a complete", "please provide", "please send a", "please upload"
)

private fun isHedged(text: String): Boolean = HEDGE_PHRASES.any { text.contains(it, ignoreCase = true) }

Every row is still written and tagged (hedged: true/false) — resume-skip needs to see it so a bad clip never gets silently re-sent to the API on the next run — but a downstream --include-hedged-gated filter drops hedged rows before they reach the actual training file. The very next batch of new clips this caught one live, automatically, no manual review needed.

From video to still frames

Gemini’s cloud API takes the raw video inline. Gemma 4 E4B’s vision tower doesn’t have that entry point in the fine-tuning tooling we’re using — it takes still images. So a second tool step (ffmpeg under the hood) samples N evenly time-spaced frames per clip and writes out training rows in the standard multimodal chat format:

{
  "messages": [
    {"role": "user", "content": [
      {"type": "image", "image": "frames/clip1/frame_001.jpg"},
      {"type": "image", "image": "frames/clip1/frame_002.jpg"},
      {"type": "image", "image": "frames/clip1/frame_003.jpg"},
      {"type": "image", "image": "frames/clip1/frame_004.jpg"},
      {"type": "text", "text": "You are a golf swing coach..."}
    ]},
    {"role": "assistant", "content": [{"type": "text", "text": "<Gemini's coaching response>"}]}
  ]
}

Worth being honest about the tradeoff here: the model is reasoning over a handful of freeze-frames, not continuous motion — closer to a coach flipping through a few printed photos than watching video.

Training the LoRA adapter

With transformers, peft and torch (MPS backend, running locally on an M5 Max), the training script freezes the entire base model and only trains a small set of low-rank adapter matrices injected into the language model’s attention projections:

tools/lora-training/train_lora.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    # Scoped to the language model only: q/k/v/o_proj also exist in the vision
    # tower, but wrapped in a custom Gemma4ClippableLinear layer there that
    # peft can't target directly.
    target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj)",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 4,538,368 || all params: 7,945,639,200 || trainable%: 0.0571

That target_modules regex was the result of a real failure: Gemma 4’s vision tower reuses the same q_proj/k_proj/v_proj/o_proj names for its own attention layers, but wraps them in a Gemma4ClippableLinear module (for QAT-style input/output clamping) that peft doesn’t know how to inject a LoRA adapter into. The fix is scoping the target to language_model specifically.

Three training runs, each on a growing, cleaner corpus:

Consistent downward trend, no instability — and v3 is the first version that never saw a hedge response during training.

One practical note: our first attempt used 8 frames per clip and 3 epochs, and a single training step took over 30 minutes on MPS without finishing — image tokens dominate the context length, and MPS lacks CUDA’s fused attention kernels. Dropping to 4 frames and fewer epochs turned a multi-hour run into single-digit minutes, with no loss in the signal we cared about for this smoke test.

Does it actually change the output?

The real test isn’t the loss number, it’s whether the generated text moved. Same clip, greedy decoding, compared across versions:

v1 adapter picked up the teacher’s **Observation:** / **Actionable Tip:** labelling convention, but the content stayed generic — “focus on a smooth transition.”

v3 adapter, on 19 clean examples, made specific, confident claims: “you are ‘casting’ (releasing the clubhead early) on the way up,” “your head stays very still,” using real golf vocabulary the way the teacher does. Structurally it now mirrors the teacher’s per-phase breakdown almost exactly.

Small dataset, so there’s still a garbled word here and there — a LoRA adapter trained on 19 examples isn’t going to be flawless — but the direction is exactly what you’d hope for: more (clean) data pulling the small model’s output toward the teacher’s style and specificity, not just a lower loss number on paper.

Merging, and a lesson in how not to verify a merge

Once you’re happy with an adapter, peft can bake it into the base weights, producing one ordinary standalone checkpoint that needs no peft at inference time:

tools/lora-training/merge_adapter.py

1
2
3
4
base_model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16)
adapted = PeftModel.from_pretrained(base_model, adapter_dir)
merged = adapted.merge_and_unload()
merged.save_pretrained(output_dir)

My first instinct for “did the merge work” was to generate from the unmerged (PeftModel-wrapped) and merged versions on the same input and diff the text. It came back MISMATCH — worrying, until we looked closer: the two outputs were structurally identical for the first few hundred characters, diverging in exact wording partway through. That’s the signature of floating-point non-associativity, not a broken merge: computing base(x) + lora_delta(x) as two separate matmuls (unmerged path) versus one fused matmul (merged path) round differently in bf16, and greedy decoding over 300 autoregressive tokens can amplify one flipped token into a fully different continuation downstream.

The real test is at the weight level:

expected = base_weight + (lora_B @ lora_A) * (alpha / r)
diff = (expected - merged_weight).abs()
# max abs diff: 0.000484, on layer 16's v_proj -- 65% of what bf16 is allowed
# to move a weight of that magnitude, and no element anywhere exceeds it

That’s the actual proof the merge is correct — a generation-based diff was simply the wrong tool for the job at bf16 precision.

One correction to the above, found later while turning that ad-hoc check into a committed script: the per-element bound alone is not a real test. Handed a merged checkpoint and the wrong adapter entirely, it still passes. The reason is that a trained LoRA delta on this model peaks around 2.7e-4, while bf16’s storage step at those weight magnitudes is ~6.1e-4 — the delta is partly rounded away by the merged checkpoint’s own dtype, so “is every element within bf16 rounding of base” is nearly as true of the wrong adapter as the right one.

What does discriminate is correlating the delta the merge actually applied (merged - base) against the delta the adapter predicts. A correct merge scores between 0.34 and 0.90 depending on the matrix and the run; a mismatched adapter scores −0.00 to 0.08 across all 132, and a merge that silently did nothing scores exactly 0. tools/lora-training/verify_merge.py now runs both checks and fails on either, with those three cases as its controls.

Converting for on-device: LiteRT-LM

The merged checkpoint (~15GB) is far too large to ship on a phone as-is. Google’s litert-torch package converts and quantizes a local HuggingFace-format checkpoint into a single .litertlm bundle for Android’s on-device runtime:

python -m litert_torch.generative.export_hf \
  ./merged/golf-swing-v3 \
  ./litert/golf-swing-v3 \
  --task=image_text_to_text \
  --quantization_recipe=dynamic_wi4_afp32

(That “original” column is the fp32 intermediate the exporter materialises, not the bf16 checkpoint on disk — which is why the language model shows 8x rather than the 4x you’d expect going bf16 → int4. The vision adapter’s 3.9x is the same fp32 baseline at int8, since the vision tower isn’t affected by the int4 recipe at all.)

It converted cleanly — but the first real on-device-style inference test (via the litert-lm CLI, which runs a .litertlm file locally without needing a phone) crashed on the very first message:

Failed to apply template: unknown method: map has no method named get (in template:238)

Turns out this is a known trap: the model’s bundled Jinja2 chat template uses a construct the lightweight on-device template parser doesn’t support, and it only surfaces at inference time, not during conversion. The fix didn’t need a re-conversion — litert-lm run accepts a template override, and Google publishes a known-compatible one alongside their own official .litertlm release of the same base model:

litert-lm run ./litert/golf-swing-v3/model.litertlm \
  --chat-template=./gemma-4-E4B-it-litert-lm/chat_template.jinja \
  --vision-backend=cpu --backend=cpu \
  --attachment=frame_001.jpg --attachment=frame_002.jpg \
  --attachment=frame_003.jpg --attachment=frame_004.jpg \
  --prompt="You are a golf swing coach..."

That worked — genuinely the first time a fine-tuned FormAI adapter ran end-to-end through the actual on-device runtime, structured output and all. But it also surfaced a second, more fundamental issue: the aggressive int4 quantization degraded generation quality, with visible repetition loops (“The swing is balanced and balanced. The swing is balanced and balanced.”) that weren’t present in the unquantized bf16 output.

A milder dynamic_wi8_afp32 (int8 weights) recipe fixed it outright. First conversion attempt hit an out-of-memory kill mid-run; a retry with a clear memory state succeeded (7.8 GiB — 8.35 GB on disk, roughly double int4’s size). Same clip, same prompt, no more repetition — clean, structured, correct golf terminology throughout. int4 isn’t viable for this model; int8 is the one that actually works, at twice the on-device footprint.

The real test: a clip the model has never seen

Every comparison up to that point was against clips the model trained on. The actual test is a clip it’s never seen — so we grabbed one more Pexels golf clip (a different camera angle entirely, shot from behind, nothing like it in the training set) and ran the base model, the int4 adapter, and the int8 adapter against it side by side.

int4 started reasonably, then collapsed into a runaway repetition loop — the same sentence, “You have a very relaxed and relaxed swing…“, repeated more than 50 times until it hit the token limit. Worse than the in-sample test, and on a genuinely novel input.

int8 stayed completely coherent: full six-phase breakdown, specific and confident (“you have a very short, ‘chicken-wing’ backswing,” “you’re ‘casting’ your club — your lead arm starts moving before your body”), correct terminology, no degeneration — on a camera angle the model had never trained on.

Base Gemma 4 E4B, no fine-tuning at all, on the same clip, was also coherent — so coherence on its own isn’t the signal fine-tuning added. The base model hedges constantly (“since these are static images rather than a continuous video, my feedback will be based on…“) and stays generic (“the takeaway appears to be initiated correctly… needs to be more aggressive”). The fine-tuned int8 model commits to specific, named diagnoses the way the Gemini teacher does. That’s the actual, verified effect of the LoRA training — not “does it produce words,” but “does it commit to a specific diagnosis” — and it held up on an input nothing in training resembled.

A truncation bug that wasn’t a truncation bug

Both the int4 and int8 responses on that held-out clip cut off mid-sentence, no closing punctuation. Reasonable first guess: a token budget limit. Two attempts at fixing it, both wrong:

1. Bumped litert-lm run --max-num-tokens at runtime — no change.
2. Reconverted the model entirely with a larger --cache_length baked in at conversion time — no change. Identical output, word for word, to the byte.

Identical output across two different length-related settings is the tell: neither was ever the actual constraint. litert-lm run --verbose confirmed it — clean exit: 0, no error, no truncation warning anywhere in the log. Unpacking the .litertlm bundle (litert-lm unpack) settled it for good: the file’s LlmMetadataProto defines its stop conditions as the model’s own turn-closing token (<|turn|>, in various forms depending on trailing punctuation) — not a length cap at all. The model is choosing to end its turn there, confirmed further by switching to sampled decoding (temperature/top-k/top-p): different content, but landing at essentially the same structural cutoff point.

So the real cause was the one I should have trusted the first time: with only 19–23 training examples, the model hasn’t reliably learned when a response is actually complete — it’s confidently closing its turn a beat early, not running out of budget. That’s a training-data-size problem, and no runtime flag was ever going to fix it. (Cheap sanity check that ruled out one alternative explanation: every single teacher response in the training set ends with proper closing punctuation, so it’s not learning the pattern from bad training data either — it just hasn’t seen enough complete responses yet.)

Growing the corpus, and the filter proving itself

Corpus went from 19 → 21 → 23 examples over a few more rounds of pulling clips, labelling them, and regenerating frames. The hedge filter — built after finding that one bad clip by hand — caught two more bad rows on its own in the next batch, no manual review:

[16/29] Labeling GOLF_SWING/pexels_6573456.mp4 (GOLF_SWING)... OK but HEDGED (1264 chars — Gemini asked for better footage instead of coaching; excluded from training by default)

That’s the filter doing exactly what it was built for — a standing gate on every future batch of clips, not a one-off fix.

A 50x speedup, and a training run that ate its own output

PyTorch’s MPS backend is a Metal translation layer bolted onto a framework built CUDA-first. MLX is Apple’s own array framework, designed from scratch around Apple Silicon’s unified memory — worth trying given where the corpus is headed: 100+ examples at PyTorch’s measured pace (~1.72 min/step, consistent across two separate runs) is a multi-hour commitment, not an evening’s experiment.

mlx-vlm claims native Gemma 4 support, including its own LoRA trainer. First run, on the same 7 examples used for the very first PyTorch smoke test, crashed immediately:

ValueError: too many values to unpack (expected 4, got 5)

A real bug in mlx_vlm’s Gemma4 vision encoder: it has two code paths for handling images, but neither expects a stack of multiple images per example — one path merges a (batch, frames, seq_len, patch_dim) pre-patchified tensor, the other expects a plain (batch, channels, H, W) single image. Our four-frames-per-example input arrives as (batch, 4, channels, H, W) — five dimensions, matching neither path. The fix is small and mirrors what the neighbouring code path already does — fold the image dimension into batch before unpacking:

.venv/lib/python3.14/site-packages/mlx_vlm/models/gemma4/vision.py (local patch)

1
2
3
4
5
else:
    if pixel_values.ndim == 5:
        b0, num_images, C, H, W = pixel_values.shape
        pixel_values = pixel_values.reshape(b0 * num_images, C, H, W)
    B, C, H, W = pixel_values.shape

With that patched, and the LoRA scope restricted to match the PyTorch config exactly (q_proj/k_proj/v_proj/o_proj only — mlx_vlm’s own default sweeps in every linear layer in the language model, MLP included, more than triple our intended parameter count) — the speed difference is not subtle:

Roughly 50-60x faster, same LoRA scope, same model, same data. A 46-step run (23 examples, 2 epochs — the same corpus the PyTorch v3/v4 runs used) finished in about a minute.

Then the actual generated text: the literal word “model”, five hundred times in a row.

Loss had collapsed from 2.56 to 0.00004 within the run — not learning, memorising. With the same nominal hyperparameters PyTorch used successfully, MLX’s ~50x speed meant “2 epochs” was a far more aggressive optimisation schedule in wall-clock terms than the number implied. Two more attempts, each ten times lower on the learning rate:

At 2e-6 — a hundredth of the PyTorch default — the adapter produced real, structured coaching output, matching PyTorch’s style closely (**Observation:**/**Actionable Tip:** formatting, “casting,” correct sequencing language) on both a training clip and the held-out clip from the section above. Tested, not assumed:

“3. Backswing (Implied) — Observation: The images show the setup and the start of the motion, but not the full backswing. Actionable Tip: Focus on a smooth, controlled takeaway… 4. Downswing & Impact — Sequence is key. The downswing should start with a slight shift of weight toward the target (ground force) before the arms and club speed up. Avoid “casting”…*”

So: a real bug found and fixed in someone else’s library, a real ~50x speedup, and a real lesson that speed without re-tuning the learning rate just gets you to a broken model faster. All three are true at once, and none of them would have shown up without actually training and generating from the result rather than trusting the loss curve.

A structured-output tax hiding in a “smarter” idea

MLX’s speed made a bigger corpus realistic, which surfaced a new question: those keyframe timestamps used for smarter frame selection (see below) were, at that point, hand-eyeballed. Gemini can find them itself — ask it for {setup, peak_wind_up, contact, finish} as structured JSON alongside the critique, in the same call, and frame extraction stops guessing.

One combined call, responseSchema covering both the critique text and the four timestamps, worked — technically. But re-labelling the whole corpus with it and diffing against the old free-text responses turned up a real regression: average critique length dropped from 1876 to 514 characters, and the markdown structure (### 1. Stance headers, **Observation:**/**Actionable Tip:** pairs) mostly vanished. Constraining part of a Gemini response to a strict JSON schema apparently taxes the other, free-text part of the same response — the model spends less effort on prose when part of its output has to satisfy a schema.

The fix was two separate calls instead of one: a plain free-text critique call — at this point byte-identical to what the app itself sends, so training data matched production input/output shape exactly — then a second, minimal call asking only for the four timestamps against a tiny schema with nothing else riding on it. (That parity didn’t survive to the end of the post: the teacher prompt later picks up an explicit length and markdown spec, described further down, which the app’s cloud prompt doesn’t have. What matters by then is that it stays byte-identical to the on-device prompt, which is the one the fine-tuned model actually infers against.) More API calls, more latency per clip — but average critique length came back to 2021 characters, slightly above the original single-call baseline, with full markdown structure restored. Re-labelling the whole corpus confirmed it wasn’t a one-clip fluke.

tools/dataset-bootstrap/…/Main.kt

1
2
3
4
val text = callGeminiForCritique(client, args.apiKey, mimeTypeFor(clip), base64, prompt)
val isHedgedResponse = isHedged(text)
val keyframeTimestamps = if (isHedgedResponse) null else
    callGeminiForKeyframes(client, args.apiKey, mimeTypeFor(clip), base64)

Keyframe-guided frames, not blind even-spacing

With real timestamps in hand, frame extraction stopped being “N frames evenly spread across the clip” and became “one frame at each named moment (setup/peak-wind-up/contact/finish), plus extra evenly-spaced frames filling that window if it’s long enough to need them”:

val targetCount = Math.round(windowDuration * framesPerSecond).toInt().coerceAtMost(maxFrames)
if (targetCount <= named.size) return named.sorted()
val filled = (0 until targetCount).map { i -> windowStart + (windowDuration * i / (targetCount - 1)) }
return (filled + named).sorted().distinctBy { Math.round(it * 20) }

A fast 1-second swing gets exactly its four named moments; a slower or more drawn-out one gets denser sampling across the actual motion instead of wasting frames on dead time before and after it. Combined with duration-scaled even-spacing as the fallback for clips without timestamps (hedged rows, mainly), every clip’s frame budget now tracks how much is actually happening in it.

Growing the corpus, and a stress test against footage that isn’t stock

Corpus went from 42 to 67 clips — 25 more Pexels golf swings, found and downloaded through the site’s own UI (direct video-page URLs trip a Cloudflare human-check that’s a hard no; the search-results grid’s own per-card download links don’t) — and re-labelled with the two-call approach above: 53 clean examples after hedge filtering, up from 33. Retraining on the bigger set (same 2e-6 learning rate, same LoRA scope) landed at a final loss around 0.5–0.7, matching the smaller run’s curve shape almost exactly.

The more interesting test wasn’t another Pexels clip, though — it was actual personal phone footage, shot on a Pixel and never touched by any part of this pipeline before. Base Gemma 4 and the newly-retrained adapter, same frames, same prompt, temperature 0, side by side:

Base model — 5 numbered sections, merging “Downswing” and “Impact” into one:

4. Downswing & Impact: The transition from backswing to downswing needs careful monitoring… Concentrate on a “lag”…

Fine-tuned adapter — 6 sections, splitting Downswing and Impact apart, matching the app’s prompt taxonomy (stance, grip, backswing, downswing, impact, follow-through) one-for-one, the way every training example was labelled:

4. Downswing & Transition: …The key is sequencing… 5. Impact: The clubhead appears to be meeting the ball relatively squarely…

That six-way split held up across three separate held-out clips, all real personal footage. What’s more telling is what didn’t happen: one of those clips turned out to be a putting stroke, not a full swing — and both models correctly recognised that and abandoned the six-category full-swing template in favour of putting-specific fundamentals (tempo, pendulum motion, a “gate drill” suggestion), landing on nearly identical structure and length. The fine-tuning added a real, content-conditioned nudge toward the training taxonomy — it didn’t teach the model to stamp that taxonomy onto everything regardless of what’s actually in the frames.

Pushing frame count further, and finding where the hardware wall actually is

Would more frames per example help — closing the gap between what Gemini’s teacher critique saw (the full video) and what the student ever sees (a handful of stills)? A controlled experiment: same 12 clips, same LoRA config, one run at the standard 4 frames, one at roughly double.

The 4-frame run trained fine. The ~10-frame run didn’t train at all:

ValueError: No trainable examples: every example has more image tokens than
max_seq_length=2048 allows. Raise max_seq_length or lower the image resolution.

Gemma spends roughly 256 tokens per image. Ten images alone is ~2560 tokens — over mlx_vlm’s default 2048-token training budget before a single word of prompt or critique text is counted. Every example in the batch got silently dropped as untrainable. --max-seq-length wasn’t exposed as a flag in our training script at all, so this had never come up before:

parser.add_argument("--max-seq-length", type=int, default=2048)
...
training_args = TrainingArgs(..., max_seq_length=args.max_seq_length)

Wiring it through and raising it to 4096 got past that check — and immediately hit a different, harder wall:

RuntimeError: [METAL] Command buffer execution failed: Insufficient Memory

Failed on the very first training step, before a single loss value printed. The 4-frame run already peaks around 40GB of the machine’s 64GB unified memory; doubling the image count pushed the compute graph for even one example past what’s available. Not a tuning problem — a real ceiling, on this hardware, for this frame count, that would need gradient checkpointing or shrinking per-image resolution to move at all. Left as an open question rather than a rabbit hole: the 4-frame keyframe-guided approach is the one that’s actually validated end to end, so it stays the default.

Getting the MLX adapter into a shippable file, without retraining

At this point there were two incompatible halves. The best adapter was trained in MLX. The merge → quantize → .litertlm path is peft/PyTorch. peft’s merge_and_unload() can’t read MLX’s adapter format, so the obvious move was retraining the whole corpus through PyTorch purely to get a mergeable checkpoint — three hours at PyTorch’s measured pace.

Before committing to that, it was worth reading what each framework actually does at merge time. mlx_vlm’s LoRALinear.fuse():

delta = ((self.scale * self.lora_b.T) @ self.lora_a.T).astype(weight.dtype)

with scale = alpha / rank. That is peft’s lora_B.weight @ lora_A.weight * (alpha / r) — the same arithmetic, with the matrices stored transposed. The only real difference is naming: mlx_vlm enumerates layers as language_model.model.layers.N…, while the HF class peft wraps calls the same layers model.language_model.layers.N….

So the “incompatibility” was a key rename and a transpose, not a numerical gap. A ~80-line script did it, and the merge verified at the weight level:

checked 132 weight matrices (skipped 36 tied/absent of 168 in the base model)
worst per-element diff: model.language_model.layers.20.self_attn.v_proj.weight
  max abs diff: 0.000227  (34.8% of its bf16 precision budget)
weakest delta correlation: model.language_model.layers.7.self_attn.q_proj.weight
  correlation: 0.3597  (floor 0.2)

Well inside bf16’s rounding, and the applied delta is unambiguously this adapter’s. Three hours of retraining avoided by reading two functions.

(The 36 skipped matrices were their own small lesson: Gemma 4 ties k_proj/v_proj across layers 24-41, so while the base checkpoint carries all 168, the merged one saves only the 132 that are independent. The first verification script asserted on all 168 and “failed” on a model that was perfectly fine — the same false-alarm shape as the generation-diff mistake earlier.)

Five errors deep: actually running it on a phone

The desktop CLI had been running this model happily for hours. Wiring the same file into the Android app took five distinct failures, each one a layer further in. It’s worth listing them in order, because the shape of the sequence is the point.

1. SentencePiece tokenizer is not found in the model. The mobile runtime wants a SentencePiece tokenizer; our bundle had a HuggingFace one. litert_torch ships a converter for exactly this — which crashed:

RuntimeError: INTERNAL: piece must not include null character.

Two real bugs behind that. First, its --normalize_tokens flag is defined and then never passed to the function it configures — lib.convert(tokenizer) ignores it entirely, so every mode silently ran as decode. Second, and the actual cause: byte-fallback tokens. Gemma’s vocab contains 256 placeholders <0x00>…<0xFF>, and in decode mode the converter calls tokenizer.decode() on each, turning <0x00> into a literal NUL byte that the SentencePiece loader rejects outright. They need passing through verbatim with type=BYTE. With both patched, plus byte_fallback = True on the trainer spec:

Not matched strictly 0/1000 pairs: 0.00%, loosely 0/1000 pairs: 0.00%

Zero mismatch across a thousand round-trip pairs — better than the ~1% the library’s own docstring warns to expect.

2. Vision encoder must have exactly one input tensor. MediaPipe’s vision calculator asserts inputs().size() == 1. Gemma 4’s vision tower takes two — pixel values and pixel position ids. Not a flag; an architecture mismatch.

I got this one wrong first. I called it “a genuine gap in the mobile SDK” and recommended either switching base models or stopping. That was premature — I’d diagnosed the symptom correctly and then stopped investigating one step too early. Google’s own LLM Inference API docs say it plainly: “The MediaPipe LLM Inference API (Android, iOS, and Web) is now in maintenance-only mode.” We were building against the deprecated runtime. The current one is LiteRT-LM — the same runtime family as the desktop litert-lm CLI that had been running this exact file, with four images, all along. It ships as com.google.ai.edge.litertlm:litertlm-android, minSdk 24, and its API takes multiple images natively.

3. Image must be preprocessed before being used in SessionAdvanced. LiteRT-LM has two layers: Session/InputData is low-level and wants pre-processed image tensors; Conversation/Content does the resizing and normalising for you. Switching to Content.ImageBytes fixed it.

4. unknown method: map has no method named get. The chat-template bug from earlier in this post, resurfacing — this time from the model’s bundled template rather than a runtime flag. Diffing the two templates makes the cause unambiguous:

Google hand-maintains a minijinja-safe variant for on-device use; the stock HF template leans on .get() throughout, which the lightweight parser doesn’t implement.

The fix worth recording isn’t the template swap, it’s how cheaply it can be applied. Re-converting the model takes ~10 minutes and had already been OOM-killed twice. But the template lives in the packaging step, not the export — and litert-lm has unpack and pack:

litert-lm unpack model.litertlm --output-dir /tmp/unpacked
# swap jinja_prompt_template in LlmMetadataProto.pbtext
litert-lm pack /tmp/unpacked/model.toml --output model.litertlm

Under a second, because pack just re-concatenates sections that already exist. A ten-minute flaky re-conversion replaced by a one-second repack.

5. Truncated output. Then it worked — genuinely coherent coaching, generated on the phone. Just cut off mid-sentence at ~1,000 characters.

This one was mine. I’d set maxNumTokens = 1000, reading it as an output cap. It’s the whole context budget, and Gemma spends ~256 tokens per image — four frames is ~1,024 tokens before a word of prompt. The model’s own metadata declares max_num_tokens: 4096; raising it to that took the response from 1,029 to 3,155 characters with the full six-phase structure intact.

The GPU question, and a manifest line that blocks it entirely

Backend.GPU() exists in the API, so: does it help? First attempt failed outright:

UNKNOWN: Can not find OpenCL library on this device

Which is odd, because libOpenCL.so and libOpenCL-pixel.so are both present under /vendor/lib64. Android 12+ requires an app to declare vendor native libraries before it can dlopen them — and comparing the two AAR manifests shows the gap:

<!-- MediaPipe tasks-genai declares these; LiteRT-LM's manifest
     contains only <uses-sdk>, so they have to be added by hand -->
<uses-native-library android:name="libOpenCL.so" android:required="false" />
<uses-native-library android:name="libOpenCL-pixel.so" android:required="false" />
<uses-native-library android:name="libOpenCL-car.so" android:required="false" />

required="false" throughout, so the app still installs and runs on devices with no OpenCL at all. Adding those three lines to the app manifest flipped the log from OpenCL not supported on this platform. Using WebGPU instead. to Loaded OpenCL library with dlopen, and the delegate from LITERT_WEBGPU to LITERT_CL. As far as I can tell this makes Backend.GPU() unusable out of the box for any app on Android 12+ — a one-line omission in someone else’s manifest, worth reporting upstream.

The payoff, same clip and prompt throughout:

Roughly 2x once LiteRT-LM’s OpenCL kernel cache is warm, with no quality difference. Not more, because of this:

Weights preparation on Gpu is disabled for PowerVR, Broadcom, Mali GPUs

The Tensor G5’s PowerVR GPU is on that exclusion list, so the weight-heavy work stays on CPU regardless — visible in the power rails, where the GPU rail sits flat around 35mW while a CPU rail swings past 1.8W. An Adreno device would likely do better. The NPU isn’t an option either: it fails to register, and litert_torch’s ahead-of-time NPU export only has vendor configs for MediaTek and Qualcomm — nothing for Tensor.

One non-obvious consequence: where OpenCL is missing, LiteRT-LM fails hard rather than falling back, and it does so at sendMessage rather than at initialize(). So the CPU fallback has to wrap the entire generate call, not just engine construction.

A measurement caution, since I got this wrong twice before checking properly: the first “GPU is 10x faster” reading came from a timer that started after the run did, and the CPU baseline I’d been quoting (20-40 minutes) came from a monitor timeout rather than an actual measurement. The honest numbers are the table above, derived from timestamps the app itself records.

Measuring before optimising, and being wrong twice

With it running, the obvious question was whether ~3.5 minutes per analysis could be improved. My instinct was that loading an 8.35GB model on every single analysis — a fresh engine constructed and closed per request — had to be the dominant cost.

Adding one log line settled it:

backend=GPU images=6 initMs=7426 genMs=205986 chars=3433 charsPerMin=1000

Model load is 7.4 seconds of 213 — 3.5%. Caching the engine would have been a 3.5% win, and I’d have built it first. Generation is 96.5%, decoding at ~4.2 tokens/sec.

Two runs at different output lengths fit a usable model: ~24s fixed (init + image prefill) + ~0.055s per character. That immediately priced the other ideas — dropping 6 frames to 4 saves about 5 seconds, which is not where the money is. The money is in generating fewer characters, because time is simply linear in them.

The obvious lever was to ask for brevity in the prompt. It worked, and it was the wrong kind of win:

**Stance:** Ensure your feet are shoulder-width apart with knees slightly flexed.

52 seconds instead of 213 — but compare what it replaced:

**Observation:** The posture appears relatively stable and athletic. The head is
down, looking toward the ball...

Told to compress, the 4.5B model threw away the observations and kept the generic tips. What’s left could have been written without watching the video at all. For a coaching app that’s exactly backwards — the observation is the part that required actually looking at the frames.

Asking for “one observation and one tip per phase, under 250 words” got both: 84 seconds, grounded observations intact. A 2.5x speedup for a prompt change.

int4: the earlier finding stands

The one remaining lever with step-change potential was int4 — halving the weights should ease the memory bandwidth that decode is bound by. Earlier in this post int4 broke the model, but that was v3 on 19 examples, and there was a decent argument that it’d behave differently now: v7 has 53 examples, and the size arithmetic says Google’s own published .litertlm is int4 (Gemma 4 E4B is ~8B params with embeddings — the peft print earlier in this post says 7,945,639,200 — so int8 lands around 8GB, int4 around 4GB, and their published file is 3.66GB).

The conversion worked cleanly and produced exactly what that predicted — 4.29 GB, with the language model at 8.0x compression against the fp32 export. Interestingly the vision tower stays int8 regardless, since vision_encoder_quantization_recipe is a separate field that defaults to int8 — so only the text side was affected.

The output was not:

| **Downswing** | The body is positioned in a transitional phase, suggesting good initiation. |
| **Impact**    | The body is positioned in a transitional phase, suggesting good initiation. |
| **Follow Through** | The body is positioned in a transitional phase, suggesting good initiation. |

Plus “relaxed and relaxed”, labels degenerating into **Stance (Stance/Balance):**, and a closing line hallucinating about “the person throwing away, the grass, the flags”. The same repetition collapse as before, on triple the corpus.

The flaw in my reasoning was the comparison: Google ships int4 for the base model. Ours is a LoRA fine-tune whose learned behaviour lives in a small delta on top of those weights — and that delta is what 4-bit rounding destroys first. More data didn’t rescue it, and probably wouldn’t. int8 stays.

Letting the teacher do the editing

The brevity prompt worked but sat awkwardly: the model was trained on one prompt and asked to obey a different one at inference. The tidier version is to ask Gemini for shorter critiques when building the corpus, so brevity is learned rather than requested — and crucially, so the compression decisions are made by the model that’s good at them.

Same idea applied to the formatting bug. The markdown was rendering as run-on paragraphs because a single newline isn’t a line break in markdown; ### headings and - bullets are. Putting that in the teacher prompt means the model emits correctly-rendering markdown natively.

Re-labelling with that prompt had an effect I didn’t anticipate:

Hedging dropped by two thirds. Asking for “one observation of what you actually see” pushes Gemini to describe the frames rather than decline over imperfect footage — so the usable corpus grew 19% as a side effect of asking for something shorter. Training loss came down to 0.328 from v7’s 0.5–0.7, and the truncated-mid-sentence problem from earlier in this post disappeared: responses now end cleanly.

The vocabulary worry didn’t materialise either. I expected shorter labels to thin out the coaching terms, but they redistributed rather than vanished — counting the share of examples that mention each term, per 100: “rotation” 52→71, “shallow(ing)” 13→30, “over-the-top” 24→31, while “lag” 20→3 and “tempo” 39→23 fell away. The concise prompt pulled Gemini toward things visible in a still frame and away from abstract timing concepts. For frame-based analysis that’s arguably the better trade.

And the honest result, measured on the same clip on the phone:

Trained-in brevity is slower than the prompt hack. Gemini’s 250-word answers average 1509 characters, the model faithfully learned that length, and it’s longer than what the small model produces when squeezed. I had assumed the principled approach would also be the faster one; it wasn’t.

What it actually bought was correctness rather than speed — markdown that renders properly, a length that’s learned rather than requested and therefore consistent, matching train and inference prompts, and 19% more training data. Worth having, just not for the reason I expected.

(One measurement footnote: the first run after pushing a new model showed initMs=68946 — 69 seconds. That’s LiteRT-LM building its OpenCL kernel cache for an unseen file. The second run was 7.8s, and produced byte-identical output, so these timings are deterministic rather than noisy. Always worth doing the second run.)

Where this leaves things

This was a smoke test of the full mechanism, not a shippable model: one activity type (golf swings), a few dozen training examples, a handful of epochs. What it did prove, for real, with real data at every step:

• Gemini can act as a teacher for distilling a much smaller on-device model
• a simple content filter on the teacher data measurably matters, not just data volume — and it keeps working unattended as the corpus grows, not just on the case it was built for
• LoRA training against Gemma 4 E4B works locally on Apple Silicon via MPS, with real, consistent loss improvement as the corpus grows (1.84 → 1.18 → 1.11 across three rounds)
• the merge step is verifiably lossless (at the weight level, not just “it ran”)
• the conversion path to Android’s on-device runtime is real and working end to end, chat-template gotcha and all, and produces genuinely usable output at int8 — verified on a clip the model had never seen, not just training data
• quantization choice is not a free lunch — int4 halves the file size and breaks the model; int8 is the one that actually works
• MLX trains this exact model and LoRA config roughly 50-60x faster than PyTorch/MPS on the same hardware — once the learning rate is dropped by two orders of magnitude to match
• constraining part of a Gemini response to structured JSON has a real cost on the free-text part of that same response — worth two API calls instead of one to avoid
• keyframe-guided, duration-scaled frame sampling beats blind even-spacing, and scales the corpus (42 → 67 clips, 33 → 53 clean examples) without changing that story
• the fine-tuning nudge is real and holds up on genuinely novel personal footage, not just more stock clips — and it’s content-conditioned, not a rigid template stamp, verified by watching it correctly not apply to a putting stroke
• more frames per example isn’t free: it broke training outright at the default token budget, and even after fixing that, it broke again against real hardware memory limits — a controlled experiment that answered its own question by failing informatively, twice
• an MLX-trained adapter can be merged by peft directly — the two frameworks compute the identical delta, so “incompatible formats” turned out to be a key rename plus a transpose, and three hours of retraining went away
• and the whole thing runs in the actual app, on an actual phone: a real golf swing from the gallery, six frames, ~98 seconds on GPU, no network — behind a settings toggle that only appears once the model file is present
• int4 still breaks this model even at triple the corpus, because the fine-tuned delta is what 4-bit rounding destroys first — the base model tolerating int4 says nothing about a LoRA on top of it
• on-device latency is almost entirely token generation (96%), not model loading (4%) — one log line killed an engine-caching optimisation I was about to write, and priced every other idea before it was tried

That last point is the one I’d have bet against at the start of the day. Getting there took five consecutive failures, each a layer deeper than the last — tokenizer, then vision encoder, then image preprocessing, then chat template, then token budget. None of them showed up on desktop; all of them were specific to the mobile runtime.

The most useful lesson isn’t any individual fix. It’s that two of the five were caused by building against the deprecated runtime, and I nearly abandoned the whole vision path over one of them, having concluded it was an unfixable SDK gap. It was — in MediaPipe, which Google’s own docs describe as being in maintenance-only mode. The supported runtime had been sitting in the same repo the whole time, and I’d been using its desktop twin all day to verify the model. Worth checking what’s actually current before concluding something can’t be done.

What’s genuinely left: a bigger and more diverse corpus (other activity types beyond golf), and model distribution — an 8.35GB file is adb push-only today, which is fine for a prototype and useless for a real user.

References

Everything the pipeline above is built on, with what each one was actually load-bearing for.

Models

• google/gemma-4-E4B-it — the base model. Source of the 4.5B-effective / 8B-total parameter split (the gap is Per-Layer Embeddings), and of the stock HF chat template with the 23 .get() calls that the on-device parser chokes on.
• litert-community/gemma-4-E4B-it-litert-lm — Google’s own on-device build of the same model. Source of the minijinja-safe chat_template.jinja used as the override, and of the 3.66GB int4 file the size arithmetic was checked against.
• mlx-community/gemma-4-e4b-it-bf16 — the MLX-format conversion of the base model, what train_lora_mlx.py actually trains against.
• Gemma 4 on LiteRT-LM — Google’s per-device benchmark numbers, and the published 3.65GB model size.

On-device runtime and conversion

• google-ai-edge/LiteRT-LM — the runtime the model is actually converted for, and the desktop CLI whose run / unpack / pack subcommands did all the verification. pack is the reason a chat-template swap costs a second instead of a ten-minute reconversion.
• LiteRT-LM documentation
• com.google.ai.edge.litertlm:litertlm-android — the Android artifact (minSdk 24). Its manifest declares no native libraries, which is what breaks Backend.GPU().
• google-ai-edge/litert-torch — export_hf for the checkpoint → .litertlm conversion, and the SentencePiece tokenizer converter that needed two local patches.
• google-ai-edge/ai-edge-quantizer — the quantization recipes behind dynamic_wi4_afp32 / dynamic_wi8_afp32.
• MediaPipe LLM Inference API — the deprecated route, and the page that states it plainly: “now in maintenance-only mode.” Two of the five mobile failures came from not reading this first.

Training

• LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., the original method.
• huggingface/peft — the PyTorch/MPS training path, and merge_and_unload().
• ml-explore/mlx — Apple’s array framework, the ~50-60x speedup.
• Blaizzy/mlx-vlm — the MLX vision-language trainer. Its Gemma 4 vision encoder needed the five-dimensional pixel_values patch, and its LoRALinear.fuse() is what showed the MLX and peft merges are the same arithmetic.

Supporting

• mitsuhiko/minijinja — the template engine LiteRT-LM embeds. Both the string minijinja and the literal error has no method named are present in liblitertlm_jni.so, which is why the stock Jinja2 template fails at inference rather than at load.
• google/sentencepiece — the tokenizer format the mobile runtime requires, and the byte_fallback / type=BYTE handling behind the piece must not include null character crash.
• <uses-native-library> — the Android 12+ declaration required before an app can dlopen a vendor library such as libOpenCL.so.
• “My Fine-Tuned Gemma 4 Loaded Fine, Then Broke on the First Message” — the same chat-template trap, hit independently and fixed by re-exporting rather than repacking.
• Pexels — the rights-cleared stock clips the whole 67-clip corpus is built from.
• joreilly/FormAI — the app itself. The pipeline lives in tools/dataset-bootstrap and tools/lora-training.