Stop eyeballing your prompts

You change a prompt. You try 2-3 cases. "Looks better." You push.

We have all done it. And we have all had the surprise a week later: "why is this working worse than before?"

The problem is not your prompt. It is that you have no way of knowing whether it is actually better.

---

The problem with "it looks fine"

When you change a prompt, three things can happen:

1. It improves the cases you tested… and breaks the ones you did not
2. It changes nothing, but you think it is better because you tested an easy case
3. It really is better — but you have no idea by how much

Without measurement, you are guessing. And guessing does not hold up at scale.

---

Two questions, two metrics

All of AI evaluation comes down to two questions:

Precision — "when my agent says yes, is it right?"

Your agent classifies support tickets as "urgent". Out of 10 tickets marked urgent, 8 really are.

Precision = 8/10 = 80%. Not bad, but 2 non-urgent customers were handled first for nothing.

Recall — "does my agent find all the real cases?"

There were in fact 12 urgent tickets. Your agent found 8 and missed 4.

Recall = 8/12 = 67%. Four urgent customers waited.

---

The trade-off

You cannot maximise both at the same time.

If your agent is ultra cautious (it marks everything as urgent) → perfect recall, but catastrophic precision. Everything is "urgent", so the word means nothing.

If your agent is ultra selective (it only marks urgent when it is 99% sure) → perfect precision, but catastrophic recall. It misses real cases.

F1 is the compromise. It is the harmonic mean of the two — a single number that tells you "overall, am I good or not".

---

There is no universal good score

This is the thing nobody tells you at the start.

Is an F1 of 0.75 good or useless? It depends.

You are detecting bank fraud → missing 25% of the fraud is catastrophic. You want very high recall, even at the cost of false positives. An F1 of 0.75 is not enough.

You are sorting blog posts into categories → getting a category wrong is not a big deal. An F1 of 0.75 is more than enough.

The right score is the one that is acceptable for your business. Not a magic number.

---

The real game changer

The biggest gain often does not come from changing model or prompt. It comes from adjusting a threshold.

Your agent says "urgent" when its confidence goes above 50%. Raise the threshold to 70% → precision goes up, recall goes down. Lower it to 30% → the opposite.

It is a dial. And without metrics, you do not even know where it is set, or which way to turn it.

---

In practice: the minimum viable setup

You do not need a complex evaluation pipeline. Just:

1. 10-15 test cases — your most frequent prompts, with the expected results
2. One simple score — even "number of cases passed out of 15" is enough to start
3. Replay before every change — does your v2 prompt pass all 15 cases? Then you can push with peace of mind

It is the equivalent of unit tests for your prompts. Not sexy, but it stops regressions.

---

The takeaway

ML is continuous optimisation under real constraints.

You will not hit 95% in real conditions. Data is imperfect, edge cases are everywhere. That is normal.

What matters: measure, iterate, and optimise for your context — not for an abstract score.