57% better, 45% faster, 41% cheaper: rebuilding TriCoach's AI coach
This is part 2. Part 1 covers how I first built the coach: one language model in a loop of plain code, calling tools until it meets a goal.
It worked, but I kept waiting. A simple question like "how did my long run go?" took around 15 seconds on average, and sometimes 20, because the model went round the loop three or four times, deciding at each turn what information to call next. I wanted the same answers, but faster and cheaper, and ideally better.
The idea: let a decision model decide
A decision model doesn't write. It is given a set of closed-ended questions, and it answers each one with probability of “yes”, ranging from 0 (no) to 1 (yes). I realized that my coach was already asking and answering questions but in a much more roundabout way, with an excessive amount of prose.
How it's put together
Jev decides, code acts, Haiku writes:
- Decide. Jev judges what the user’s message needs using up to 24 questions. Is it about a finished workout? Does it ask for a change? Is the user ill or tired? Does the answer require a visualization?
- Choose. A small truth table in code turns those judgments into tool calls. An unsure answer counts as yes, because loading a little extra is cheap and missing something costs a round trip.
- Load. The app runs every tool call at the same time, before the language model starts.
- Write. Haiku 5.5 gets one request including all outputs from the tool calls. It writes its reply in a single response, requiring less reasoning. As a fail safe, if it needs more information to provide an answer, Haiku is granted one more request for additional tool calls so that the answer is never blank.
- Check. Jev checks that the reply addresses what the user asked (again, with a probability).
Nothing else changed: the same prompt, the same tools, the same guardrails.
How I tested it
I wrote 100 questions of the kinds I actually ask: how did my workout go? Can I move a session to tomorrow? I’m sick/traveling, how do I adjust my plan? How should I fuel my workout? What is the science behind the training plan? I ran each question through the old coach and the new version using the same test athlete. Claude Opus 5.5 then graded every set of answers blind, in random order, using the test athlete’s data.
Here is a snippet of the results:
| Today's coach | Haiku decides | Jev decides | Jev, fast | Jev, fast vs today's coach | |
|---|---|---|---|---|---|
| Quality (1 to 10) | 4.75 | 6.35 | 7.05 | 7.45 | 57% higher |
| Average time | 15.2 s | 14.9 s | 10.2 s | 8.4 s | 45% faster |
| Average cost per question | $0.0029 | $0.0030 | $0.0024 | $0.0017 | 41% cheaper |
| LLM requests per question | 3.7 | 4.2 | 1.9 | 1.2 | 68% fewer |
| Timeouts | 1 | 0 | 0 | 0 | 0 vs 1 |
- Today's coach: one model in a loop, deciding at each turn what to read.
- Haiku decides: the same design as the new coach, with Haiku making the judgments instead of Jev.
- Jev decides: Jev decides what to read, but the model still works in a loop instead of loading all tool calls at once.
- Jev, fast: one pass, less thinking, the reply streamed as it's written. The one I kept.
Takeaway
Break down the generation of the agent's output into decisions, reasoning and writing the answer. Match each task to the right model. In my case, the decision model is what makes the answer better, faster and cheaper.
If you train, coach or build agents, I'd like to compare notes.
Thank you for reading.