All writing

Essay 02 · AI product design

Designing AI that admitswhat it doesn’t know.

Hedging isn’t honesty. In Qualifi.ai the AI has to take a position on advice — and is never allowed to invent a fact.

Written by
Parth Raval
Published
Reading
3 minutes
Topic
AI product design
Related
Qualifi.ai

Qualifi.ai is the AI career product I have been building. At its centre is Compass, a coach with memory you can read and edit, résumé tailoring, and interview practice scored against a rubric.

The hardest design problem in it isn’t a screen. It is what the AI says when it doesn’t know.

01

Hedging is not honesty

The lazy fix for an AI that might be wrong is to make it hedge. “It depends on many factors.” “Ultimately, only you can decide.” It reads as humble, and it is useless. Someone asking whether to take an offer already knows it depends. They came for a view.

So in Qualifi the rule runs the other way. Compass has to take a position — “I’d take it if…”, “I wouldn’t, because…” — and the eval suite fails a reply that doesn’t. The stock hedges are written into the tests as failures.

Uncertain and vague are different things. An uncertain answer sounds like: “I’d lean towards the startup, but I don’t know how long their funding lasts — ask that before you sign.” It commits, it names exactly what it doesn’t know, and it tells you how to find out. That is the shape I want every answer to have.

02

Never fill a gap with something plausible

The opposite failure is worse. Ask a model to tailor a résumé and it will happily add “increased revenue by 30%”, because résumés like that look right. Ask about pay and it will give you a number with total confidence.

So the line I drew is simple.

Decisive about advice. Never about facts it wasn’t given.

Every figure in a tailored résumé is checked against the original, and a reply that names a salary or an employer it made up fails. The model is free to have an opinion; it is not free to have a source it doesn’t have.

src/lib/ai/evals · assertions

The lineCheckResult
“Great question — here’s the thing”noFillerOpenings fail · opens with filler
“you can definitely do this”noBannedWords fail · banned voice word
“this next journey”noBannedWords fail · “journey”
“you could earn $120k there”noFabricatedSpecifics fail · invented a salary
“It really depends on a lot of factors.”takesAPosition fail · hedges, no view
“Honestly, I’d take the offer — the upside is real.”takesAPosition pass · commits to a view
Fig. 01 — Lines the Qualifi evals fail, and one they pass. The strings are fixtures from the test suite.

03

Make uncertainty a step, not a shrug

Often the most honest answer a career tool can give is “neither of us knows yet”. The design question is what happens next.

In onboarding, “I’m not sure yet” is a real answer with its own path, not an error state. In Crossroads, someone who doesn’t know what they would do instead gets two or three directions, each with a real experiment they can run in about a week. The uncertainty doesn’t go away. It gets something to do.

04

Design for the day the AI isn’t there

Models go down, keys expire, budgets hit their cap. Every AI surface in Qualifi has a fallback a person actually sees, and those fallbacks are checked on every run with zero model calls. The model-graded quality cases — twelve of them, from “two very different people must get different plans” to résumé fidelity and refusals — run only when a key is present, and skip cleanly instead of pretending to pass.

I made the cheap checks the important ones on purpose. Using a model to judge a model is useful, but if the gate needs a model to run, the day the model is unavailable is the day nothing gets checked.

05

Admit it to your own team first

Qualifi’s promise is that Compass remembers you. In one mode today it doesn’t keep every turn yet, and that is written in the project’s risk file in plain words, with the fix next to it — not left for a user to discover.

An AI product that admits uncertainty to its users has to start by admitting it to the people building it.

06

What I’d tell another designer

  • Decide where the AI must commit and where it must say “I don’t know”. Write both down.
  • Turn the voice and honesty rules into tests. A style guide nobody runs is a wish.
  • Treat “not sure” as a path with its own design, not an empty state.
  • Design the fallback first. It is what people see on the worst day.

— ParthKitchener–Waterloo · 28 Oct 2026