Shrinking an AI coach onto a phone: a plain-English tour of LoRA fine-tuning

FormAI is a Kotlin Multiplatform app that watches a video of your golf swing and tells you what to fix. Normally it uploads the video to Google’s Gemini API and gets coaching text back.

It can now also do that entirely on the phone, offline, in about 98 seconds — using a much smaller model that was taught to imitate Gemini for this one specific job.

This post describes how that’s put together. It assumes no background in fine-tuning, so it explains the concepts as it goes. There’s a much more detailed engineering log of the same work in a separate post if you want stack traces and measurements.

The problem in one paragraph

Gemini runs in a data centre. The models small enough to run on a phone are, out of the box, noticeably worse — ask one to coach a golf swing and you get feedback that could have been written without watching the video (“the takeaway appears to be initiated correctly… needs to be more aggressive”). What we want is what the big model gives you: “you’re ‘casting’ your club — your lead arm starts moving before your body.” A specific, named fault.

So the goal isn’t “run a model on a phone.” It’s “make a small model behave like a big one, for one narrow task.”

The pipeline

Six stages, each of which is a tool or script in the repo:

1. Label a folder of golf clips by sending them to Gemini, producing “teacher” critiques.
2. Filter out clips the teacher couldn’t actually critique.
3. Extract four still frames from each clip at the moments that matter.
4. Train a small LoRA adapter on the resulting (frames → critique) pairs.
5. Merge that adapter permanently into the base model’s weights.
6. Quantize and convert the result into the single file Android’s on-device runtime loads.

The rest of this post walks through those.

1. Gemini as the teacher

The technique here is distillation: use a large model as a teacher to generate training data, then train a small student model to imitate it. The appeal is that you never need a human to write thousands of expert golf critiques — Gemini already produces them, so you just need to collect enough.

A command-line tool (tools/dataset-bootstrap) walks a directory of clips, sends each to Gemini, and writes the response out as a row of JSON. It’s resumable, so an interrupted run never re-spends an API call on a clip it already labelled.

The corpus is 67 short, rights-cleared stock clips from Pexels rather than anyone’s personal footage.

Each clip actually gets two calls. The first asks for the critique in plain text. The second asks only for four timestamps — setup, top of the backswing, contact, finish — against a small JSON schema. Splitting these matters: when a single response has to satisfy a strict schema and produce good prose, the prose measurably suffers. Keeping them apart means the critique call is exactly the free-text request the app itself makes.

The prompt asks for one observation and one actionable tip per swing phase, in markdown, under 250 words:

You are a golf swing coach.
Analyze this swing video and provide concise, actionable feedback on stance,
grip, backswing, downswing, impact, and follow-through.
For each phase give one observation of what you actually see, then one
actionable tip, formatted exactly as:

### Stance
- **Observation:** ...
- **Actionable Tip:** ...

Keep the whole response under 250 words, with no preamble or closing summary.

Baking the length and formatting into the teacher’s prompt — rather than asking the small model to be brief at runtime — means brevity is something the student learns rather than something it’s told. It also means the prompt used during training is byte-for-byte the prompt used on the phone, which keeps the model on familiar ground when it runs for real.

2. Filtering the teacher’s output

Sometimes Gemini declines to critique, because the clip genuinely doesn’t show enough:

“I cannot provide a complete analysis of your stance, grip, or the full arc of your swing… please provide a face-on view and a down-the-line view.”

That’s a sensible response, but a terrible training example. The student model has no way to know the refusal was justified — it just learns that sometimes the correct output is to ask for a better video. Train on enough of those and you get a model whose main skill is declining to help.

So every response is scanned for hedging phrases ("can't see", "please provide", "unable to", and so on) and tagged. Tagged rows are still written to disk, so the resume logic never re-sends them, but they’re held out of the training file.

Of the 67 clips, 4 are filtered out this way, leaving 63 usable examples. It’s the least glamorous component in the pipeline and one of the most valuable.

3. From video to frames

Gemini’s API takes video directly. The tooling for fine-tuning the small model takes still images. So each clip is reduced to four frames.

Which four matters. Spacing them evenly across the clip breaks down on a 27-second video where the swing itself occupies two seconds — you get four frames of someone standing still. Instead, the timestamps from the teacher’s second call drive the extraction: one frame at each named moment, with extra evenly-spaced frames filling in if the window is long enough to warrant them.

val targetCount = Math.round(windowDuration * framesPerSecond).toInt().coerceAtMost(maxFrames)
if (targetCount <= named.size) return named.sorted()
val filled = (0 until targetCount).map { i -> windowStart + (windowDuration * i / (targetCount - 1)) }
return (filled + named).sorted().distinctBy { Math.round(it * 20) }

A fast one-second swing gets exactly its four named moments; a slower one gets denser sampling across the part where something is actually happening.

Worth being upfront about the compromise: the teacher watched the video, the student only ever sees four photographs. It’s closer to a coach flipping through printed stills than watching someone swing.

4. What LoRA is, and how it’s configured

Here’s the part that sounds harder than it is.

A model is a very large pile of numbers — “weights”. Gemma 4 E4B, the model used here, has about 8 billion of them. Traditional fine-tuning adjusts all of those numbers, which means holding the entire model plus a lot of bookkeeping in memory at once. On a laptop, that’s usually a non-starter.

LoRA (Low-Rank Adaptation) is a shortcut. You freeze the model completely — every one of those 8 billion numbers stays exactly as it was — and bolt on a small set of extra numbers alongside it. Training only ever touches the extras.

The analogy: rather than rewriting an encyclopaedia, you write margin notes. The encyclopaedia is untouched, the notes are tiny by comparison, and when you read, you read both together.

The ratio is what makes this practical. The training script reports:

trainable params: 4,538,368 || all params: 7,945,639,200 || trainable%: 0.0571

4.5 million numbers out of 7.9 billion — 0.057% of the model. That’s why this runs on a laptop at all, and why the resulting adapter is a few megabytes rather than several gigabytes.

The adapter is scoped to the attention layers of the language model only, deliberately leaving the vision tower alone:

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj)",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

Training runs on MLX, Apple’s own array framework, which is built for Apple Silicon rather than adapted from something built for NVIDIA hardware. A training step takes roughly 2 seconds, against about 1 minute 43 seconds for the equivalent PyTorch step on the same machine — so two passes over the 63-example corpus take a few minutes rather than the best part of a day.

One setting is worth calling out because it doesn’t transfer between the two frameworks. The learning rate controls how big an adjustment to make at each step, and the MLX setup uses 2e-6 — a hundredth of PyTorch’s usual default. Because MLX gets through the same nominal number of passes so much faster, the conventional value is far too aggressive here and drives the model into memorising the training set rather than learning from it. Lower is correct on this setup, not merely safer.

(A footnote on the model’s name: “E4B” means roughly 4.5 billion effective parameters despite the raw count of about 8 billion. Gemma uses Per-Layer Embeddings, where some weights are large lookup tables that don’t all need to be active at once. That’s why a nominally 8B model counts as phone-sized.)

5. Merging

After training you have two artifacts: the untouched base model, and your adapter. For running on a phone you want one file, so the adapter is merged — its adjustments are applied to the base weights permanently and saved as a new standalone model that needs no adapter machinery at inference time.

Because MLX and the PyTorch tooling store adapters differently, a bridging script rewrites the MLX adapter into the format the merge tool expects. The two frameworks compute an identical adjustment; the difference is naming and matrix orientation, not arithmetic.

The merge is then verified at the level of the weights themselves, which takes two checks. The first confirms every adjusted weight equals base-plus-adjustment to within the precision the file format can represent. The second confirms the change the merge actually applied correlates with the change this specific adapter predicts — necessary because LoRA’s adjustments are small enough that the first check alone will happily pass when handed the wrong adapter entirely. Together they cover 132 weight matrices:

checked 132 weight matrices (skipped 36 tied/absent of 168 in the base model)
worst per-element diff: model.language_model.layers.19.self_attn.v_proj.weight
  max abs diff: 0.000243  (39.5% of its bf16 precision budget)
weakest delta correlation: model.language_model.layers.0.self_attn.q_proj.weight
  correlation: 0.3419  (floor 0.2)

(The 36 skipped matrices aren’t missing — Gemma shares some attention weights across its later layers, so they don’t exist as independent tensors to check.)

6. Quantizing and converting

The merged model is around 15GB, far too big to ship.

Quantization shrinks it by storing each weight with less precision — a whole number from 0-255 (“int8”, one byte each) rather than a full-precision value. It’s the difference between recording every price to the cent and to the nearest euro: you lose accuracy, you save an enormous amount of space.

Google’s litert-torch handles both the quantization and the conversion into .litertlm, the single-file bundle Android’s on-device runtime loads:

python -m litert_torch.generative.export_hf \
  ./merged/golf-swing-v8 \
  ./litert/golf-swing-v8 \
  --task=image_text_to_text \
  --quantization_recipe=dynamic_wi8_afp32

That produces an 8.35GB file.

The recipe choice is deliberate. There’s a more aggressive int4 option that halves the size again, and Google ships an int4 build of the base model that works fine — but it isn’t viable for a fine-tune. The useful behaviour here doesn’t live in the large base weights; it lives in the small adjustments LoRA layered on top, and coarse rounding destroys small numbers first. Int4 damages precisely the part the training created, producing repetition loops and invented detail. Int8 is the recipe that works, at double the footprint.

On the phone

The Android side uses LiteRT-LM (com.google.ai.edge.litertlm:litertlm-android). The app pulls frames from the chosen video, scaling the count to the clip’s length, and hands them to the runtime along with the same prompt the teacher was given:

val engine = Engine(
    EngineConfig(
        modelPath = modelFile.absolutePath,
        backend = backend,
        visionBackend = backend,
        maxNumTokens = MAX_TOKENS,
        maxNumImages = images.size,
        cacheDir = context.cacheDir.absolutePath
    )
)

A few details that matter in practice:

• maxNumTokens is the whole context budget, not the response length. Each image costs about 256 tokens, so several frames consume a meaningful share before any text exists. It’s set to 4096, the maximum the model itself declares.
• GPU first, CPU as fallback. The GPU path is faster, but where OpenCL is unavailable the runtime fails rather than degrading gracefully — and it fails when generating, not when initialising. So the fallback wraps the entire generate call, not just engine construction.
• The GPU path needs three manifest lines. Android 12+ requires an app to declare vendor-provided native libraries before it can load them, and the LiteRT-LM package doesn’t declare any:

<uses-native-library android:name="libOpenCL.so" android:required="false" />
<uses-native-library android:name="libOpenCL-pixel.so" android:required="false" />
<uses-native-library android:name="libOpenCL-car.so" android:required="false" />

• The model file is copied on by hand (adb push) into the app’s external files directory. There’s no download flow yet, so the on-device toggle only appears in settings once the file is actually present.

What it does

On a Pixel 10 Pro XL, with no network, an analysis takes about 98 seconds on the GPU.

The performance profile is lopsided in a useful way:

backend=GPU images=6 initMs=7426 genMs=205986 chars=3433 charsPerMin=1000

Loading the 8.35GB model: 7.4 seconds. Generating: 206 seconds. Model loading is about 4% of the total, so caching the engine between analyses would buy almost nothing. Time scales almost perfectly with the number of characters produced, which is why the length instruction lives in the teacher prompt — generating less text is the only lever that meaningfully moves the number.

The exact same file also runs on a laptop, through the litert-lm command-line tool, which is a useful way to calibrate what “on-device” actually costs you. On an M5 Max MacBook Pro, same prompt, same four frames:

The laptop’s GPU is roughly 14x faster than the phone’s, and its CPU alone still beats the phone’s GPU by about 6x. Repeated runs produced byte-identical output, so these are deterministic rather than noisy.

The first run is slower in every case because the runtime compiles and caches the model’s kernels the first time it sees a given file — on the laptop that cache is a 4.8GB file written next to the model. On the phone it’s dramatic: 71 seconds of the first run’s 158 is cache building, against 8 seconds once warm. Generation time is identical either way. So a user’s very first analysis takes over 60% longer than every one after it.

And the number that puts all of this in perspective — the cloud path the app already had, same clip, same phone, now that its prompt matches the on-device one exactly:

Of those 12 seconds, 3.5 is the phone compressing the video and 8.4 is the round-trip to Google. Running locally costs about 8x the latency. That’s the real price of the offline, nothing-leaves-the-device version.

One detail worth noting for anyone trying this: on the Mac the GPU path runs through WebGPU on Metal, whereas on Android it needs OpenCL. Those are the same --backend=gpu flag doing quite different things underneath.

As for quality, the comparison that matters is against the same base model with no fine-tuning, on footage that never touched the pipeline. The base model produces five sections, merges downswing and impact together, and hedges about being shown still images. The fine-tuned model produces six sections matching the app’s exact taxonomy — stance, grip, backswing, downswing, impact, follow-through — with a specific observation in each. That six-way split is exactly what every training example looked like.

The most interesting result came from a held-out clip that turned out to be a putting stroke rather than a full swing. A model that had merely memorised a template would have stamped the six full-swing categories onto it anyway. Instead it recognised what it was looking at, dropped the template, and talked about tempo, pendulum motion and a gate drill. That’s the difference between learning a format and learning a task.

That said, beating the untrained base model is a low bar. Compared against Gemini itself on the same clip, with both now given byte-identical prompts, the gap is clear. The structure is indistinguishable — both produce six phases, an observation and a tip each, near-identical length. The content is not:

Across all six phases the small model describes what something looks like, while Gemini names a fault and its consequence. They occasionally contradict each other outright, and Gemini is the more reliable witness — it watched twelve seconds of video where the small model saw five still frames.

So the fine-tuning reproduced the teacher’s shape very well and its judgement only loosely. For a model running offline on a phone that’s a genuine achievement, but it’s worth being clear that on-device here buys privacy and offline use, not better or faster answers.

One rough edge that hasn’t gone away: the on-device response still stops mid-sentence, without a closing full stop. With 63 training examples the model hasn’t reliably learned what a finished answer looks like — and no runtime setting fixes it, because it isn’t running out of room, it’s deciding it’s done.

Where it stands

This is a working prototype rather than a shipped feature. It covers one activity, it’s trained on 63 examples, and the 8.35GB model has to be side-loaded — not something you can reasonably ask a user to download.

What’s in place is the full mechanism, end to end: a big model generating training data, a filter keeping that data honest, a 0.057% adapter trained on a laptop in a few minutes, a verified merge, an int8 conversion, and a phone producing structured coaching offline in 96 seconds.

Whether it’s worth using is a separate question, and the honest answer today is no — the cloud path is 8x faster and gives better answers. What the on-device path buys is that the video never leaves the phone and it works with no signal. That’s a real feature, just not the one the latency numbers flatter.

Two lessons worth carrying off, beyond this app. Matching a teacher’s format is far easier than matching its judgement, and it’s easy to confuse the two if you only ever compare against an untrained baseline. And nearly all the on-device cost is generating text rather than loading the model — except on the very first run, where cache building dominates.

The remaining work is corpus breadth — other activities beyond golf — and a real distribution story for the model file.