Skip to content
Back to blog

Prompt Engineering in Production: What Actually Works

Production prompt engineering: 6 patterns for reliable LLMs — chain-of-thought, domain few-shot, JSON schema, fallback prompts, git versioning, and a golden eval harness.

The tutorials make prompt engineering look like a one-shot affair: write a prompt, get a result, ship it. Production tells a different story.

At scale, prompts fail in ways you can’t anticipate from 10 test cases. The output format drifts between model versions. Edge cases return malformed JSON that crashes your parser. The prompt that worked perfectly last week now fails on a new class of inputs that your users just started submitting. And the engineer who wrote the original prompt left no record of what they were trying to achieve.

This is post 6 in the AI pillar series. Previous posts covered AI automation for business, AI agents for internal tools, when RAG is worth building, choosing the right LLM, and LLM costs in production. This one is about the six patterns that make the difference between a prompt that works in demos and one that stays reliable at scale.

Pattern 1: Chain-of-thought for complex decisions

The problem: For tasks requiring multi-step reasoning — triage, classification, analysis — asking the model to jump directly to a conclusion produces inconsistent results.

Before:

Classify this support ticket as: billing, technical, account, or other.
Ticket: {{ticket_text}}

After:

Classify this support ticket. First, identify what the user is asking for
in one sentence. Then identify which category best matches that need:
billing (payments, invoices, pricing), technical (bugs, errors, performance),
account (login, permissions, profile), or other.

Output your reasoning in 2-3 sentences, then output CATEGORY: <category>.

Ticket: {{ticket_text}}

Chain-of-thought has a real cost: more output tokens. The tradeoff is worth it on decisions where a wrong classification triggers a real downstream consequence — routing to the wrong team, firing the wrong workflow, surfacing the wrong response to the user.

Pattern 2: Few-shot examples with real domain data

The problem: Generic instructions fail on domain-specific nuance. “Summarize this” means something completely different for a legal contract, a customer support ticket, and a technical specification.

Before: A prompt with instructions only.

After: A prompt with 2–4 real examples drawn from your actual data, showing exactly what a correct output looks like for your domain.

The examples don’t need to be exhaustive — they need to be representative. Ideally they come from the edge cases that the baseline prompt handled poorly. Three well-chosen real examples from production data consistently outperform ten generic synthetic ones.

One essential rule: never use examples that contain real user data without proper anonymization. Use anonymized versions of real patterns, not the original records.

Pattern 3: Structured JSON output schema

The problem: Free-form text output is unparseable at scale. Models vary their output format subtly across inputs, especially when the structure is complex.

Before:

Summarize the key risks from this contract.

After:

Analyze this contract and return a JSON object with this exact structure:
{
  "risk_level": "low" | "medium" | "high",
  "key_risks": [{"risk": "...", "clause": "...", "severity": "low"|"medium"|"high"}],
  "missing_clauses": ["..."],
  "recommendation": "..."
}

Return only valid JSON with no additional text.

Contract: {{contract_text}}

Better yet: use a tool/function-calling interface if your provider supports it — it enforces schema compliance at the API level and eliminates the need for JSON extraction from surrounding prose. Claude’s tool use, OpenAI’s function calling, and Gemini’s JSON mode all solve this problem structurally rather than via instruction.

Pattern 4: Fallback prompts for low-confidence outputs

The problem: Some inputs are genuinely ambiguous. A single prompt can’t return both a confident result and signal that it’s uncertain — unless you build that signal in.

Before: One prompt, one output, no confidence signal.

After: Add a confidence field to your JSON schema:

{
  "category": "billing",
  "confidence": "high" | "medium" | "low",
  "reasoning": "..."
}

Then route low-confidence outputs to a fallback path: a second more expensive model for re-evaluation, a human review queue, or a safe default response. This creates a quality tier in your pipeline — most outputs go through the cheap fast path; the uncertain ones get a second look.

The cost overhead is small (only a fraction of requests land in the fallback path) but the quality impact is disproportionate. Customers rarely notice correctly handled easy cases; they always notice incorrectly handled hard ones.

Pattern 5: Versioned prompt files in git

The problem: Prompts evolve over time, but without version control you lose the ability to audit changes, roll back regressions, or understand why a prompt changed.

The fix is unglamorous but essential: treat prompts as code. Store them as files in your repository — prompts/support-classifier/v2.txt, prompts/contract-risk/v3.json — and version them with the same practices you apply to code: meaningful commit messages, PR review for significant changes, a changelog entry when behavior changes.

This practice unlocks two things that matter in production:

  1. Regression tracking. If output quality degrades after a prompt change, you can git blame and revert.
  2. A/B testing. Running two prompt versions in parallel and comparing their outputs against an eval suite is only possible when prompt versions are identifiable artifacts.

The second one leads directly to pattern 6.

Pattern 6: An eval harness with 20+ golden test cases

The problem: Prompt changes are made by feel — “this version seems better” — with no systematic measurement. The change ships, quality degrades in ways that only show up in production complaints weeks later.

The fix: before deploying any prompt change, run it against a set of golden test cases — inputs with known expected outputs — and measure pass rate. Twenty or more cases is the minimum to detect regressions reliably; 50–100 is better for high-stakes prompts.

The eval harness doesn’t need to be complex:

  1. A CSV file with columns: input, expected_output, expected_category
  2. A script that runs each input through the new prompt and compares outputs
  3. A pass/fail threshold — e.g., 90% of cases must pass to ship

This turns “I think this prompt is better” into “this prompt passes 94% of test cases vs. 87% for the previous version.” That’s the signal you need to make prompt changes with confidence rather than hope.

Tools like promptfoo and LangSmith formalize this pattern. For early-stage teams, a 50-line Python script with your golden cases in a CSV file is enough to start.

The one thing that ties all six patterns together

None of these patterns are complex in isolation. The compounding problem is that most teams add them reactively — pattern by pattern, prompted by a specific production failure. Chain-of-thought after the classification regression. The eval harness after the prompt change that silently degraded quality for a week before anyone noticed.

The teams that avoid that cycle build these patterns in from the start, treating prompt engineering with the same discipline as any other production system: versioned, observable, tested, with fallback paths for failure cases.

If you’re building AI features and want to get the foundation right before you’re chasing failures in production, a fractional CTO can help you structure the system from the start. Book a free intro call — no commitment required.

This is post 6 in the AI pillar series. Start with AI automation for business, continue through AI agents for internal tools, RAG: when it’s worth it, choosing the right LLM, LLM costs in production, and then this article. Next: AI observability in production — the 5 metrics and tooling that make LLM features visible when something goes wrong.