I built an LLM that writes dashboards. Here's how I stopped trusting it.

I built a tool that answers plain-English questions about an e-commerce dataset by generating a dashboard on the fly. Ask "which categories generate the most revenue," and an LLM writes a query, runs it, and renders a chart. That part is not hard. Every AI demo does this.

The hard part is trusting the output enough to ship it. This post is about the three things that made the difference: a contract the model can't talk its way around, a retry loop that uses its own failures as instructions, and an eval harness that catches regressions before a user does. It ends with a production bug that had nothing to do with any of that and everything to do with a 100MB file nobody told the deploy platform about.

Code: github.com/techbyhemant/olist-dashboard-agent. Live: olist-dashboard-agent.vercel.app.

The model never touches the UI

The obvious way to build this is to let the model write HTML, or JSX, or a chart library call, directly. I didn't do that, because there's no way to check that output before it reaches a user. "Looks plausible" is the only bar generated markup can clear, and plausible is not the same as correct.

Instead the model emits one thing: a typed JSON object called a DashboardSpec. Each widget in it owns the SQL that fetches its own data:

{
  "title": "Revenue by Category",
  "widgets": [
    {
      "type": "tile",
      "title": "Total revenue (delivered)",
      "metric": "revenue",
      "sql": "SELECT ROUND(SUM(oi.price), 2) AS revenue FROM order_items oi JOIN orders o USING (order_id) WHERE o.order_status = 'delivered'"
    }
  ]
}

That single decision, SQL as data instead of layout, is what makes the whole thing checkable. A JSON contract can be validated. Generated HTML can only be eyeballed.

A contract the model can't talk its way around

The spec is checked against a Zod discriminated union before anything runs. Four widget types, each .strict() so unknown keys are rejected outright, not silently dropped:

export const WidgetSchema = z.discriminatedUnion('type', [
  TileWidget,
  TileGroupWidget,
  ChartWidget,
  TableWidget,
]);

The part I spent the most time on isn't the shape validation, it's the SQL itself. The model is instructed to write read-only SELECTs. Instruction is not enforcement, and I was not going to hand a model-generated string to a database connection and hope. So every sql field runs through a safety gate before it's accepted:

export function checkSafeSelect(sql: string): SafetyCheck {
  // strip comments and string literals first, so keywords inside
  // quoted data can't trip the checks below
  const statements = stripped.split(';').map(s => s.trim()).filter(Boolean);

  if (statements.length > 1) {
    return { ok: false, reason: 'only a single statement is allowed' };
  }
  if (!/^\s*(with|select)\b/i.test(statements[0])) {
    return { ok: false, reason: 'query must start with SELECT or WITH' };
  }
  if (FILE_FUNCTIONS.test(stripped)) {
    return { ok: false, reason: 'file-access functions are not allowed' };
  }
  return { ok: true };
}

This is allowlist-first on purpose. I didn't write a denylist of dangerous keywords to chase forever. I required the statement to structurally be a single SELECT or WITH, which excludes INSERT, DROP, ATTACH, COPY, and everything else in one move. It also blocks read_csv/read_parquet/glob and friends, so the model can't reach outside the tables it's been given. This check runs twice: once as a Zod refinement (so a violation becomes a validation error the retry loop can act on), and once as a hard assert immediately before execution. Defense in depth, not defense in one place.

The loop that uses its own mistakes as instructions

A spec passing validation doesn't mean it's correct. The model can write syntactically valid JSON where the SQL aliases a column as total while the widget's metric field says revenue. Nothing in the schema catches that, because the schema doesn't know what the SQL returns. Only running it does.

So every widget's SQL actually executes against DuckDB, and the result columns are checked against what the spec claims they'll be. If that fails, or the spec fails Zod validation, or the model returns malformed JSON, the specific error gets fed back to the model as a retry prompt. Not "try again." The actual message: result has no column "revenue" (got: total).

● Generating dashboard spec…
● Spec valid: "Total revenue" with 1 widget(s).
▲ SQL execution failed — tile "Total revenue": result has no column "revenue" (got: total)
● Retrying with the error fed back to the model (attempt 2/2)…
● Spec valid: "Total revenue (corrected)" with 2 widget(s).
● All widget SQL executed and returned rows. ✓

One retry, then it fails openly rather than rendering something wrong. I capped it there deliberately. An unbounded retry loop hides how often the first attempt actually works, and that number is the one I actually care about.

Every step in that log is streamed to the browser as it happens, not buffered and dumped at the end. The self-correction is a visible part of the product, not debug output I'm hiding from the user.

Three layers, cheapest first

None of this is worth much without a way to catch regressions before a user does. I run three eval layers, ordered so the cheap deterministic ones gate every commit and the expensive one only runs when I ask for it.

Layer 1 — schema validity. No model, no database. Feeds known-good and known-bad specs through the Zod contract and checks that malformed ones get rejected with a usable reason. This is where the SQL safety gate gets its own direct tests: DROP TABLE, multi-statement injection via ;, read_csv_auto calls, all asserted rejected.

Layer 2 — spec correctness against real data. Real DuckDB, no model. Runs every widget's SQL from a golden spec, checks the actual returned rows match the declared shape, and separately drives the full generate → validate → run → retry loop against a deterministic mock generator that fails on purpose and succeeds on retry. This is the layer that verifies the loop itself works, independent of whether the LLM behind it is any good that day.

Layer 3 — LLM-as-judge. Real model, gated behind RUN_LLM_EVALS=1 so it never runs in CI by default and never costs anything unless I explicitly ask. For each golden question, the pipeline generates a real dashboard, and a second model call scores 1 to 5 how well it answers the question, given the JSON spec:

You are a strict evaluator of analytics dashboards. Given an operator
question and a dashboard spec (JSON), rate 1-5 how well the dashboard
ANSWERS the question. Judge relevance of widgets/metrics to the question,
not styling.

The judge scores relevance, not correctness, because Layer 2 already proved the numbers are real. What Layer 2 can't tell you is whether the model picked the right three widgets for what was actually asked.

The bug that had nothing to do with any of this

I deployed to Vercel. The homepage loaded fine. Every single query returned a 500.

Not our error handling either, an actual framework crash page. Our route wraps everything in a try/catch that always returns a 200 with { ok: false, error }, so this was throwing before our code ran at all.

vercel logs --expand --since 10m --level error

Error: Failed to load external module @duckdb/node-api-...:
Error: libduckdb.so: cannot open shared object file: No such file or directory

Here's the part that isn't obvious until you hit it. @duckdb/node-api's native binding, duckdb.node, is a thin wrapper. It doesn't contain the actual database engine. At load time it dynamically links a separate ~100MB libduckdb.so sitting in the same package directory:

$ otool -L duckdb.node
  @rpath/libduckdb.dylib (compatibility version 0.0.0, current version 0.0.0)

Vercel's deploy step traces the module graph to decide which files make it into the serverless function. That tracing works by statically following require() and import() calls. A native dlopen of a sibling file on disk is invisible to that. The small addon got traced and included. The 100MB engine it depends on did not, and got silently left behind, no build warning, no error, just a missing file discovered the first time a real request ran.

The fix was one line, forcing the whole binding directory into the function bundle explicitly rather than trusting the tracer to find it on its own:

outputFileTracingIncludes: {
  "/api/generate": [
    "./node_modules/@duckdb/node-bindings-linux-x64/**",
  ],
},

I mention this because it's the kind of failure that no amount of unit testing catches. Layer 1 and Layer 2 evals were green the entire time this was broken in production, because they run locally, where the binding was never missing. The only way I found it was reading the actual runtime logs of the actual deployed function after hitting the actual endpoint. Evals tell you the logic is right. They don't tell you the thing you shipped is the thing you tested.

What I'd build next

A tool/semantic-metrics layer as an alternative to letting the model write raw SQL at all, which is the safer default for anything beyond a portfolio project. Retry-efficacy tracking, so I know the first-attempt success rate by failure class instead of just watching it anecdotally. And I'd want the judge layer running on every deploy, not just on demand, once I trust the cost enough to leave it on.

If you want to poke at it, the live demo runs against a 4,000-order sample of the real dataset, and the full source has all three eval layers, the retry loop, and the SQL safety gate.