I built an AI SaaS without knowing what my LLM was really doing

In 2024 I built Luny-AI, a SaaS that generated exportable Figma mockups from a plain prompt. You describe your interface, the LLM generates the design, you export it to Figma.

It worked. People used it. But there was one thing I could not see at all: what happened between the user's prompt and the final result.

I had basic logs. Sentry for crashes. But on the LLM side? Nothing. And today, digging into how to actually master LLMs, I realise everything I was missing.

---

What I could not see

Why did some generations take 10 seconds and others 3?

I called the Claude API, I received the response, I measured total time. But between the prompt and the response, I had no idea what was going on.

Was the model "thinking" longer on certain prompts? Was it the complexity of the requested mockup? The length of the prompt? A latency issue on the API side?

I could not answer. I had a global stopwatch, not a breakdown per stage.

Which call exactly had an error?

When it crashed, Sentry told me there was an error. Fine. But when the LLM answered without crashing and returned something unusable — an incoherent mockup, a broken layout, misplaced components — I had nothing.

No HTTP error. No crash. Just a bad result. And no way to understand why the model produced that rather than something else.

How much did each generation cost?

I knew how much I paid the Claude API at the end of the month. But I did not know how much each individual generation cost. Did a complex prompt ("generate a dashboard with 6 widgets, charts and a sidebar") cost 10x more than a simple one ("generate a login form")? Probably. But I did not have the numbers.

How do you improve a "meh" result?

This is the one I missed the most. When the LLM answered but the result did not satisfy me, I tweaked the prompt on instinct. I tried one wording, then another, then another. No metric. No score. No objective way of knowing whether v2 of my prompt was better than v1.

---

What I could have seen

Today I know there are tools and patterns to trace all of this. Here is what I could have put in place on Luny-AI.

Traces, not just logs

A log tells you "something happened". A trace tells you the whole story of a request, stage by stage.

On Luny-AI, generating a mockup went through several logical stages (prompt analysis, layout structuring, component generation, assembly). Each stage could have been a span in a trace:

Mockup generation "analytics dashboard"
│
├─ Prompt analysis            [120ms]
├─ Layout structuring         [350ms, 800 tokens]
├─ Component generation       [1200ms, 2400 tokens]  ← the slowest
├─ Final assembly             [200ms, 600 tokens]
│
Total: 1870ms, 3800 tokens, ~$0.04

Suddenly I would see that component generation takes 65% of the time. That is where to optimise, nowhere else.

A cost per generation

By tagging every API call with the number of tokens consumed, I could have seen:

Prompt "login form"            →   800 tokens  →  $0.008
Prompt "dashboard, 6 widgets"  →  4200 tokens  →  $0.042
Prompt "full landing page"     →  6800 tokens  →  $0.068

And above all, I could have spotted the anomalies: a simple prompt consuming as much as a complex one means the model is looping or generating noise. That is a signal to change the prompt.

Quality metrics

This is what I missed most. When a result "is not good", that is subjective. With metrics, it becomes measurable:

Is the response valid JSON?               → yes/no
Are all requested components present?     → 4/6
Does the layout match the asked structure? → yes/no

This is what is called a rubric score — a list of criteria that can be checked automatically. No human needed to grade every generation. You define your rules, you score automatically, and you watch your success rate move as you change your prompt.

A golden set to iterate on

Instead of testing on instinct, I could have created 10-15 reference prompts with their expected results. On every prompt change, I replay those cases and compare the scores. If it goes up, the change is good. If it goes down, I roll back.

It is the equivalent of unit tests for your prompts. Not sexy, but it stops regressions.

---

What I learned

Observability is not just logs

A log tells you "error at 14:23". A trace tells you "the component generation span took 4 seconds because the prompt contained 6 nested widget descriptions, which produced 4200 tokens and cost $0.04".

The difference is depth. The log tells you there is a problem. The trace tells you where, why, and how much it costs.

Every bit of LLM "thinking" has a price

When the model takes longer, it is usually because it is generating more tokens. More tokens = more expensive. If your prompt is ambiguous, the model "hesitates" and produces more text to cover the possible cases.

A precise, structured prompt → fewer tokens → faster → cheaper. Observability shows you that correlation in numbers.

The real gain is measured iteration

Changing a prompt on instinct is like debugging without logs. You can get there, but it is slow and random.

With metrics (rubric score, response time, cost per generation, share of valid results), every change is measurable. You know whether your change improved things or made them worse. You iterate in 10 minutes instead of 2 hours.

---

What I would do differently

If I rebuilt Luny-AI today, here is what I would add on day 1:

Traces per generation — every call to Claude tagged with the type of mockup, the number of tokens, the response time, the cost. Not just a "call succeeded" log.

An automatic rubric score — is the output JSON valid? are all requested components there? is the layout coherent? Scored automatically on every generation.

A golden set of 15 prompts — the most frequent cases and the most edge ones. Replayed on every prompt change to check that nothing regresses.

A simple dashboard — cost per day, share of valid results, average generation time. Not a NASA dashboard, just the 3-4 numbers that matter.

Sanitisation from the start — only where needed

---

The takeaway

I built an AI product that worked. But "it works" is not enough when you cannot see what is happening inside.

Observability is not something you add when you have time. It is what lets you understand, improve and control what your LLM does, instead of crossing your fingers on every API call.