AI Observability in Production: What to Measure and How

You shipped an LLM feature. It works. Then, three weeks later, users start complaining that something feels off — responses are slower, some answers seem wrong, costs spiked. You open your dashboards and find: nothing. Your LLM calls are invisible.

This is the observability gap most teams discover too late. They instrument their databases, their APIs, their queues — and leave the AI layer completely dark.

This is post 7 in the AI pillar series. Previous posts covered AI automation for business, AI agents for internal tools, when RAG is worth building, choosing the right LLM, LLM costs in production, and prompt engineering in production. This one is about closing the observability gap: the five metrics that matter, the tooling that captures them, and the one dashboard view every on-call engineer needs.

Why LLM monitoring is different from regular API monitoring

Standard API monitoring tells you if your service is up and responding fast. LLM monitoring has to answer a harder question: is the model doing what it’s supposed to do?

A request can return HTTP 200, within acceptable latency, with a valid JSON payload — and still be completely wrong. The classification was off. The summary missed the key point. The answer was hallucinated. None of those failures show up in your existing infrastructure metrics.

LLM observability requires a second layer: measuring not just whether the model responded, but what it returned and how much it cost to get there.

Metric 1: Latency p95 (not average)

The average response time is a lie. What matters for user experience is p95 — the latency at the 95th percentile. If your average is 1.2 seconds but your p95 is 8 seconds, one in twenty users is waiting eight seconds for a response. That’s the number that drives abandonment.

Track latency at two levels:

• End-to-end latency: from the moment the user triggers the LLM call to when the response is rendered. This includes your preprocessing, the API round trip, any retrieval steps, and postprocessing.
• TTFT (time to first token): for streaming responses, this is what users actually perceive as “fast.” A call with 4-second total latency but 0.3-second TTFT feels instant because the user sees text immediately.

Alert on p95 end-to-end latency crossing a threshold (a reasonable starting point: alert at 3×your baseline). Separately track TTFT for streaming endpoints.

Metric 2: Token throughput

Tokens in, tokens out — and the ratio between them. This metric tells you three things:

1. Input token count per request: if this spikes, your prompts or retrieved context is growing unexpectedly (a common sign of a bug in a RAG pipeline that’s retrieving too many chunks).
2. Output token count per request: if this grows, the model is generating longer responses than expected, which raises latency and cost simultaneously.
3. The ratio: a sudden shift in input/output ratio often signals a prompt change having unintended effects on response verbosity.

Token throughput is also your capacity planning metric. At your current request rate, how many tokens per minute are you consuming? At what request rate do you hit provider rate limits?

Metric 3: Error rate by prompt version

This is the metric most teams skip, and the omission causes real damage.

LLM errors aren’t just HTTP 5xx responses from the provider. They include:

• Parse errors: the model returned text that failed your JSON schema validation
• Timeout errors: requests that exceeded your client-side timeout threshold
• Content refusals: the model declined to respond (common when prompts drift toward policy edges)
• Empty or truncated outputs: the response was valid but empty, or was cut off mid-sentence

Track error rate as a percentage of total requests, segmented by prompt version. This lets you detect regressions immediately after a prompt change rather than days later in user complaints.

The version tagging is critical. Every LLM call should carry a prompt_version attribute in its metadata. Without it, you can’t distinguish “this prompt version has a 3% error rate” from “something is wrong with my entire LLM integration.”

Metric 4: Semantic drift detection

This is the most advanced metric on the list, and the most valuable for long-running LLM features.

The problem: even when nothing changes on your side — same prompt, same model, same inputs — LLM responses drift over time. Providers update model weights. The model’s behavior on edge cases shifts subtly. What was a consistent, reliable output in month one starts producing occasional outliers in month three.

Semantic drift detection works by embedding LLM outputs and tracking the distribution of those embeddings over time. When outputs start moving away from your baseline distribution, it’s a signal worth investigating — either the model changed, or the nature of user inputs changed, or a recent prompt edit had side effects you didn’t catch in your eval harness.

Practical implementation: sample 1–5% of production responses, embed them with a cheap embedding model (text-embedding-3-small from OpenAI costs fractions of a cent per 1,000 responses), and plot the centroid and variance of each day’s sample against your baseline. A drift in centroid position or an increase in variance warrants a manual review of the sampled outputs.

Metric 5: Cost per request

You saw the aggregate cost number in last month’s invoice. But do you know the cost per successful request? Per user? Per feature?

Cost per request answers the question that your LLM economics depend on: at what scale does this feature become unsustainable? If your current cost is $0.008 per request and your target CAC is $200, the LLM cost is trivial at 10,000 requests. At 1,000,000 requests, it’s $8,000 — that’s a line item worth managing.

Track cost per request at the feature level, not just the API level. One application may have five different LLM-powered flows with very different cost profiles. Without feature-level attribution, you can’t prioritize optimization work.

Tooling: what to actually use

Three options, each with a different tradeoff:

Langfuse — open-source, self-hostable, excellent trace-level visibility. Captures every prompt and completion, supports prompt versioning natively, has a good UI for browsing traces and drilling into errors. The best choice for teams that want visibility without vendor lock-in. Free tier is generous; self-hosting is straightforward on a small VM.

Phoenix by Arize — open-source, focused on LLM evaluation and embedding analysis. Particularly strong for semantic drift detection and comparing prompt versions against an eval dataset. Good complement to Langfuse if you want dedicated evaluation tooling.

Custom ClickHouse table — for teams with high request volumes and specific cost attribution needs. Log every LLM call as a row: timestamp, feature, prompt_version, model, input_tokens, output_tokens, latency_ms, error_type, user_id. ClickHouse handles billions of rows at low cost and gives you arbitrary aggregation with sub-second query times. The tradeoff is setup effort: you build the instrumentation, the aggregation queries, and the dashboards yourself.

The one dashboard view every on-call engineer needs

If you can only build one dashboard, make it this: a single view showing, for the last 24 hours, broken down by prompt version:

• p95 latency
• Error rate (with breakdown by error type)
• Token input/output ratio
• Cost per request

When something goes wrong at 2am, this view tells the on-call engineer within 30 seconds whether the problem is a provider-side latency spike, a prompt regression, an input distribution shift, or a cost anomaly. Without it, they’re blind.

Start small, instrument early

You don’t need to build all five metrics before you ship. The right sequence:

1. At launch: error rate + latency p95. These catch the most impactful failures.
2. After first month: cost per request + token throughput. These inform optimization.
3. At scale: semantic drift. This becomes relevant when you have enough volume to sample meaningfully and enough runtime to establish a baseline.

The critical thing is to start. Every week an LLM feature runs uninstrumented is a week of production failures that go undetected until a user files a support ticket.

If you’re building AI features and want to get the observability foundation right from the start — rather than retrofitting it after the first production incident — a fractional CTO can help you design the monitoring layer before it becomes urgent. Book a free intro call — no commitment required.

---

This is post 7 in the AI pillar series. Start with AI automation for business, continue through AI agents for internal tools, RAG: when it’s worth it, choosing the right LLM, LLM costs in production, prompt engineering in production, and then this article.