We built a grader for our AI. Then we caught it being unfair.

You ask an AI for career advice. It answers in two seconds, sounds sure of itself, and moves on. The part nobody mentions: you have no way to tell whether the advice was any good. Most of the time, neither does the company that built it. They ship the answer and hope.

We didn't want to hope.

So a while back we built a second system whose only job is to grade the first one. Every answer Careersy AI gives you, a separate model scores out of 10 the moment it lands, on one question. Did this actually help you. Was it specific to your situation, did it leave you with a real next move, or was it confident filler.

Here is the catch we couldn't get past. A grader is just one opinion. What if the grader is the one that's wrong.

This is the part that stayed with me from years on the hiring side. I watched capable people get marked down by systems that were never built to see them properly. A keyword filter. A rushed recruiter. A rubric measuring the wrong thing. So when we built a grader of our own, the honest question was never "is the AI any good." It was "is our grader the broken filter now."

We ran two checks. A stronger, more expensive model re-graded a batch of real answers, blind, with no idea what our grader had said. And I sat down and graded the same batch myself, by hand, the way I grade a coaching call.

I scored those answers an average of 8.5 out of 10. Our grader scored the same answers 6.4.

It was marking good work down by two full points. Not noise. A steady, two-point harshness, worst on exactly the detailed, specific answers that should have scored highest. Our grader had quietly become the thing we exist to fight. A filter calling capable work a fail. Doing to good answers what a bad ATS does to good candidates.

So we rebuilt it.

The old grader handed back one gut-feel number, 1 to 10. Gut-feel numbers drift. The new one scores five specific things, each out of 2, then adds them up:

• Did it answer what you actually asked.
• Was it built around your situation, not generic advice.
• Did it leave you with a clear next move.
• Was it tight, with no padding.
• Was it complete.

Five times two is ten. A score you build, not a vibe you reach for. We also told it, in plain words, to stop being stingy. A genuinely useful answer is an 8 or a 9, not a 6 with a frown. Then we ran the same test again. The new grader landed at 8.6 against my 8.5. The gap was gone.

The grader and I finally agree on what good looks like. It only took rebuilding the grader.

That should have been the win. It came with a problem we had not thought to look for.

A grader that agrees with you is no use if it does not show up. And about one answer in eleven, ours did not. It would look at a perfectly good reply and hand back nothing. A garbled number. A blank where a score should sit. The answer reached the user fine, but our own record of whether it was any good came back empty, and we would not have known unless we went digging.

If you have ever fired an application into a careers page and heard nothing for three weeks, you know this from the other side. The worst rejection is the one that never happened, because nobody read the thing. A score that never gets made is that same silence, and this time it was ours.

So we shut the gap two ways. First, we boxed the grader in. We changed the way it has to answer so the only thing it can physically produce is a real score. A high one or a low one, but never a mess. Not garbled, not blank. We did not patch that failure. We made it impossible.

Second, we put a gauge on the gauge. A second reading whose only job is to watch the grader and count how often it fails to return a score. It sits on the same dashboard as everything else, so a blind spot like this one cannot sit there unseen again.

Then we shipped it and watched real conversations come through. Before the change, the grader was quietly losing about one answer in eleven. After it, the first batch of live answers came back clean. Every one scored. The number we had been missing was zero.

None of this is something you will ever see on screen. That is sort of the point. "Careersy AI improves over time" is not a line we hope is true. It is a loop we run on purpose. Every answer gets graded. The grader itself gets checked against a human, and watched for the times it fails to grade at all. When it is wrong, we say so and fix it. A tool that grades its own work honestly, including the result it didn't want, is one that actually gets better. The rest just say they do.

The next answer you get from Careersy AI is being graded the second it lands. Not because we have to. Because you are using it in a real job search, and a confident-sounding non-answer is not a small miss — it costs you time you do not have.

---

Under the hood

• Every answer, scored. A fast model grades 100% of responses right after you get them. Internal dashboard. You never see it, and it never delays your reply.
• The calibration. A stronger model re-scored 118 real answers blind. I hand-graded a stratified sample of them myself, also blind. My average came out at 8.53. The old grader gave the same answers 6.35, about 2.2 points harsh. The stronger model landed at 8.18, close to me, which told us the grader was the outlier, not the human.
• The rebuild. Five dimensions (relevance, specificity, actionability, voice, completeness), scored 0 to 2 each and summed, plus a clear instruction to stop being stingy. Agreement on the good, okay or bad call went from about 59% to about 80%. The average moved to roughly 8.6, in line with mine.
• The silent failure (the second fix). The rebuilt grader still went quiet on about 1 in 11 answers (8 of 90 in one live window), returning a garbled or blank score we would not have caught without going looking. We switched it to a constrained format, so the only thing it can output now is a valid score, and added a second reading that tracks the grader's own miss rate on the same dashboard. On the first batch of live answers after the change, the miss rate was zero. The guarantee is the format, not the sample size.
• Still honest about the edges. One area, LinkedIn post drafts, still runs slightly strict. Many of those turns are mid-draft, so they genuinely score lower on completeness. This is a loop, not a finished thing. Old and new scores are versioned so they never get blended, and the calibration is a script we re-run whenever we suspect drift.