Cloud Guru LLM Engineering Production Framework

// When should you apply the LLM Engineering Production Framework?

Use this skill whenever you are architecting, auditing, or debugging an LLM-powered system and need a principled engineering checklist rather than generic advice. Trigger it when moving from a working demo to a production system, when a live system is misbehaving, or when scoping a new AI feature end-to-end.

// What do you need before applying this framework?

• System descriptionrequired
  What the LLM system does — its purpose, users, and expected traffic pattern.
• Current pain point or build stagerequired
  Which layer is causing problems or needs to be designed: foundations, agent architecture, RAG pipeline, production hardening, framework selection, or enterprise integration.
• Model(s) in use or under consideration
  Provider, model name, and any known constraints (context window, pricing tier, reasoning mode availability).
• Knowledge corpus or data sources
  For RAG or enterprise scenarios: describe the documents, APIs, or databases the system must draw on.
• Cost and latency targets
  Acceptable per-request cost and p95 latency, if known.

// What core principles underpin production LLM engineering?

// How do you apply the LLM engineering framework step by step?

1. 1

   Establish foundation: model mechanics and the generation call

   Confirm you understand next-token prediction run in a loop. Write a production-grade API call: read the key from the environment (never hardcode), pass model, max_tokens, and a messages list with explicit roles. Always check stop_reason after every call — 'end_turn' means natural completion, 'max_tokens' means the response is truncated and must not be parsed blindly. Log usage (token counts = your bill).

2. 2

   Set sampling parameters deliberately

   Reach for temperature first for the creativity-versus-consistency tradeoff. Use top_p to tame the tail. Use top_k as a blunter alternative. Set max_tokens as a hard ceiling; set stop_sequences to end generation precisely. Never assert byte-for-byte equality on a live model — even temperature=0 is not a guarantee of identical output across batches or model updates. Check whether the target model even accepts a temperature parameter; frontier models may expose 'effort' settings instead.

3. 3

   Design prompt structure with roles and few-shot examples

   System message sets the model's job and rules. User messages carry the task. Assistant messages (pre-filled) are worked examples. Zero-shot: just ask. Few-shot: show one or two solved examples and let the model copy the pattern. Chain-of-thought: ask the model to reason step-by-step before answering — accuracy jumps on harder problems. Anti-patterns to eliminate: vague instructions, contradictory rules, and overstuffed prompts that bury the one thing that actually matters.

4. 4

   Implement structured output with the Constrain → Validate → Retry loop

   Choose the right tier: JSON mode (fast, only promises JSON-shaped output), strict JSON schema (shape enforced at decoding), or forced tool call (validated typed arguments plus an action in one move). Always validate at runtime against your Pydantic or Zod schema — the model can return well-formed JSON that is semantically wrong. On failure, send the error back to the model as explicit feedback. Never parse a streaming object token-by-token; wait for the object to complete.

5. 5

   Implement the Perceive → Think → Act loop for agent behaviour

   The agent perceives current state (conversation + observations), emits a tool call block (tool_use), your code executes it, and feeds a tool_result back. The loop runs again. Apply the ReAct pattern: force the model to write a short Thought before each Action, then read the Observation before the next Thought. This grounds each step in what actually came back from the world rather than a plan committed to before any data existed. Cap iterations — unbounded loops are a cost and safety hazard.

6. 6

   Add Reflection (Generate → Critique → Revise) for quality-sensitive outputs

   After producing a draft, make a second call whose job is to find flaws: logical gaps, unmet constraints, hallucinated citations. Feed the draft plus critique back in and request revision. Cap revisions at 2–3 passes — diminishing returns and rising cost appear quickly. Strongest signal comes from external checks (run the test, hit the type-checker) not self-critique alone, because a model blind to an error in the draft is often blind to it in the critique too.

7. 7

   Design the agent's memory architecture across the four lifetime layers

   Scratch pad (short-term, one run): append observations and intermediate results to the context window — nothing is remembered unless you deliberately carry it forward. Session memory: the running message history. Long-term memory (cross-session): write to a durable store (file, database, vector store) and retrieve at each step — this is retrieval in disguise. Apply all your RAG tooling to your agent's own past. Design questions: what to write, when to fetch, and how much. Never let the model write secrets or sensitive personal data into persistent memory files.

8. 8

   Build the RAG pipeline with the five-step checklist

   1) Chunk on meaning (headings, paragraphs, code blocks) not fixed character counts — chunking a PDF by fixed character length and slicing tables mid-sentence is the most common RAG bug. Keep metadata. 2) Retrieve with hybrid search: dense embeddings (semantic) plus BM25 (exact keywords, product codes, error strings) fused with Reciprocal Rank Fusion. 3) Rerank the short list (top 20–50) with a cross-encoder. 4) Evaluate both layers separately: retrieval (recall@K, MRR) and generation (faithfulness/groundedness, answer relevance). When an answer is wrong, suspect retrieval first — most RAG bugs are retrieval bugs, not generation bugs. 5) Pick and tune your ANN index: HNSW for best recall-latency tradeoff when RAM allows; IVFPQ when dataset is huge and RAM is the binding constraint.

9. 9

   Implement prompt caching with stable-first prompt structure

   Everything stable goes first (frozen system prompt, deterministic tool list, large reference document). Everything volatile goes last. Mark stable chunks with a cache_control block. Watch three usage fields: cache_read_input_tokens (billed at ~10% of normal), cache_creation_input_tokens (small write premium), and plain input_tokens (full price). If read count stays at zero across identical requests, an upstream element is silently invalidating the prefix — hunt it down. Toggling reasoning on/off or changing the thinking budget invalidates the cache prefix below it.

10. 10

    Route requests with a cost-aware router

    Do not send every request to your most expensive model. Route simple tasks (classification, extraction, short replies) to a small fast cheap model; reserve the flagship for genuinely hard work. Routing wrong is expensive in both directions: a hard query on the weak model gives a bad answer; an easy query on the flagship burns money for no gain. Tune the threshold against your own eval set. For reasoning models, reserve them for problems that are genuinely hard and whose answers you can verify — instrument the cost so a routing mistake doesn't surprise you at end of month.

11. 11

    Harden with guardrails, prompt-injection defences, and sandboxed code execution

    Input side: screen with moderation, deny-list, and schema validation before the model sees anything. Output side: validate against schema and content policy; refuse to act on failures. Allow lists beat deny lists whenever you can enumerate acceptable inputs. For prompt injection (the SQL injection of the LLM era): treat retrieved content as data not commands, keep a clear privilege boundary, require human confirmation for irreversible actions, and assume every byte the model reads could be hostile — there is no complete fix, only layers of risk reduction. For code execution: sandbox in an isolated container or microVM, cap CPU/memory/disk and wall-clock time, and default to no network access.

12. 12

    Instrument observability with OpenTelemetry GenAI semantic conventions

    Trace every request as a tree of spans: top-level request → retrieval calls → model calls → tool invocations, with timing, token counts, and cost at every node. Log actual prompts and completions — for a non-deterministic model, the input and output are the bug report. Apply redaction and retention policies (not a fire hose into plain-text logs). Use OpenTelemetry GenAI semantic conventions so traces sit in the same system as the rest of your services. Pin a version of the spec. Sample smartly at scale.

13. 13

    Build a three-layer evaluation system

    Layer 1 — Offline eval set: curated real inputs with known good answers, run on every prompt/model/index change. Freeze a golden set; treat a score drop as a bug. Layer 2 — LLM-as-Judge: use a strong model to grade open-ended outputs against an explicit rubric (RAGA-style frameworks automate generation metrics). Layer 3 — Regression testing: every fixed failure joins the set forever. Layer 4 — Red teaming: adversarial suite covering jailbreaks, prompt injection payloads, system-prompt extraction, disallowed content. Automate it; treat every real incident as a new test case. Pick benchmarks whose task shape matches your system (SWEBench for coding agents, TauBench for tool-using customer-service agents) — a leaderboard number is not a grade for your app.

14. 14

    Apply batch API for throughput-tolerant workloads

    Submit large asynchronous jobs (overnight document processing, bulk embeddings, eval generation) via the batch API at roughly half the normal price. Results come back in any order within a turnaround window (up to 24 hours). Key each request with your own identifier and match on output — never by position. Do not wait synchronously; submit, poll for completion, then retrieve results.

15. 15

    Choose a framework for the hard part of your problem, not the demo

    LangChain: glue and integration for linear pipelines. LangGraph: controllable stateful agent workflows with cycles, branching, and human-in-the-loop checkpoints — reach for it when your flow needs to go backwards. AutoGen: multi-agent conversation where debate improves the answer — requires firm termination rules and a turn budget or agents loop forever. CrewAI: role-based crews for fast prototyping when the problem maps onto a team of specialists. LlamaIndex: data/retrieval spine for search and Q&A over private corpora. The rule: choose a framework for the hard part of your problem, not for the part that looks good in the demo. Adopt one to solve specific pain you actually feel; stay close enough to the underlying API that you can always drop down when you have to.

16. 16

    Expose reusable tools via MCP servers and package expertise as Skills

    Build an MCP server once (tools = verbs the model can invoke, resources = nouns it can read) and every MCP-aware client can use it without a rewrite — MCP is the USB-C of model tooling. Keep each tool narrow, name it clearly, document its arguments thoroughly (the description is the entire interface the model sees). Scope permissions, validate inputs, never expose a tool you wouldn't hand a stranger. Package team expertise as Skills: a folder with a skill.md (YAML front matter naming the skill and its trigger description, followed by detailed instructions). The description is the trigger — the model loads the full skill body only when a task matches it. Progressive disclosure keeps context lean.

17. 17

    Apply the enterprise LLM pattern for SAP/ERP environments

    Default to the side-by-side pattern: build the agent as a separate application on BTP, reach into SAP through OData and RAP. This keeps the core clean and upgrade-safe. Ground every model answer in real SAP data — a plausible-looking wrong number is a financial statement problem. The LLM drafts (extraction, classification, natural language to query); deterministic SAP logic commits (posts documents, moves money). Use HANA Cloud vector engine for RAG grounding. Never autodeduplicate master data — flag and present to a human. For S/4HANA migration: use agents to scan custom code for clean-core readiness, explain violations, and suggest cloud-safe rewrites — but the strategic green-field vs. brownfield decision is not delegated to the model.

// What does this framework look like applied to real systems?

// What mistakes should you avoid when building production LLM systems?

• Never hardcode your API key — let the SDK pull it from the environment.
• Always check stop_reason after every generation call — if it is 'max_tokens', your response is truncated and parsing it (e.g., as JSON) will fail.
• Temperature zero is stable but not a guarantee of identical output — never write a test that asserts byte-for-byte equality on a live model.
• The newest frontier models may not accept a temperature parameter at all — check the docs for the model you are actually calling.
• The API is stateless — the model remembers nothing between calls. If you want multi-turn conversation, you must resend the entire history on every single request.
• Chunking a PDF by fixed character length slices tables and sentences in half — respect document structure or half your retrieval problems appear before a single query runs.
• You must embed your query with the exact same model you embedded your documents with — mixing models silently wrecks results.
• Putting a timestamp or request ID at the top of your system prompt invalidates the entire cache prefix below it and your cache hit rate silently drops to zero.
• Toggling reasoning/thinking on or off, or changing the thinking budget, invalidates the cache prefix — a setup that was cheap because of caching can suddenly cost full price.
• Most RAG bugs are retrieval bugs, not generation bugs — when an answer is wrong, check whether retrieval even fetched the right document before blaming the model or prompt.
• The lost-in-the-middle effect is real — models attend well to the start and end of long contexts but reliably miss information buried in the middle. More context is not automatically better.
• Indirect prompt injection is the dangerous case — your agent fetches a web page or email and text in that untrusted content is addressed to the model. There is no complete fix; layer defences and assume every byte the model reads could be hostile.
• Running generated code in your own process is a non-starter — a single bad command can wipe files or exfiltrate secrets. Always sandbox with hard resource limits and no network access by default.
• Treating a public leaderboard number as a grade for your app is wrong — a model that tops SWEBench may be mediocre at your retrieval-heavy support bot. Always validate on your own eval set.
• Adding multiple agents too early multiplies token cost, latency, and failure modes — try a single well-designed agent with the right tool surface first. Add agents only when you can point to concrete parallelism or context isolation a single loop cannot give you.
• In batch API results come back in any order — never match results by position, always use your own per-request identifier.
• Streaming a JSON object token-by-token and parsing incrementally breaks on partial objects — wait for the complete object or use a parser built for partial input.
• In multi-user agent systems, scope memory per user — otherwise one person's agent will read another's notes.
• Never let the model write secrets or sensitive personal data into persistent memory files — those files get replayed into future contexts.
• In SAP/enterprise contexts, never autodeduplicate master data — flag candidates and present to a human. A plausible-looking wrong number is a financial statement problem, not a bug.

// What key LLM engineering terms should you know?

Next Token Prediction Run in a Loop

The complete mechanical description of how every language model generates text: predict the most likely next token, append it, repeat until a stop condition is met. The single most important intuition because it explains why fluency and correctness are different axes.

Auto-regressive

The generation mode where each token is conditioned on all the tokens before it, which is why responses stream out word by word and why longer inputs are quadratically more expensive to process with attention.

Stop reason

The field in every API response that tells you why the model stopped generating. 'end_turn' means natural completion; 'max_tokens' means the response was cut off mid-thought. Always check it before trusting or parsing the output.

Temperature

The master sampling control. Near zero, the model almost always takes the single most likely token — focused and repetitive. Higher values flatten the probability distribution, letting surprising tokens through. The primary creativity-versus-consistency dial.

Top P (Nucleus Sampling)

A sampling method that keeps only the smallest set of tokens whose cumulative probability adds up to P, trimming the unlikely tail. Use it to tame the tail after setting temperature.

Prompt Caching

A provider feature that stores a processed stable prefix (system prompt, fixed documents, tool list) so repeated calls can serve it from cache at roughly one-tenth the normal token cost. Requires a strict prefix match — everything stable first, everything volatile last.

Context Window

The total token budget for one API call: prompt + tools + history + the model's own answer, all counted together. The real skill is using it efficiently, not just maximising its size.

Reasoning Models

Models trained to generate a long internal chain of thought before committing to an answer, controlled by an effort setting or a thinking budget. Genuinely better on hard verifiable problems (math, code, multi-step logic); wasteful and slow on easy ones. Route accordingly.

Lost-in-the-Middle Effect

The documented tendency of language models to attend well to content at the start and end of long contexts but miss information buried in the middle. More context is not automatically better; position and relevance of what you include matters.

Perceive → Think → Act Loop

The core agent architecture: the model perceives current state (conversation + observations), thinks and decides on an action (emitted as a tool call), your code executes it, the result is fed back as a tool result, and the loop repeats.

ReAct (Reason + Act)

An agent prompting pattern that interleaves Thought (visible reasoning about current state and next need), Action (one tool call), and Observation (what the tool returned), grounding each step in real data rather than a plan committed to before any results existed.

Generate → Critique → Revise Loop

A reflection pattern where, after producing a draft, a second model call evaluates the draft for flaws, and the draft plus critique are fed back in for revision. Cap at 2–3 passes; external checks (test runners, type checkers) produce stronger signal than self-critique.

Scratch Pad

Short-term memory that lives inside a single agent run — observations, intermediate results, and notes appended to the context window so later steps can see earlier ones. Dies when the run ends.

RAG (Retrieval-Augmented Generation)

The pattern of embedding a user query into a vector, searching pre-embedded document chunks for closest matches, and handing the top results to the model as context before it generates an answer. Keeps knowledge in an index you can edit without retraining.

Hybrid Search

Running dense vector search (semantic, catches paraphrase and synonyms) and BM25 lexical search (catches exact keywords, product codes, error strings) simultaneously, then fusing the two ranked lists with Reciprocal Rank Fusion.

Reciprocal Rank Fusion

A method for combining two ranked lists (e.g., dense and lexical search results) by rank position rather than raw scores, side-stepping the problem that the two systems produce scores on totally different scales.

Cross-Encoder Reranker

A model that reads the query and each candidate document together and produces a true relevance score far more accurately than first-pass retrieval. Expensive — only run on the top 20–50 candidates, never the whole corpus.

HNSW (Hierarchical Navigable Small World)

A graph-based ANN index where each vector links to its neighbours and search greedily walks the graph from coarse top layers down. Best recall-latency tradeoff; memory-hungry at hundreds of millions of vectors.

IVFPQ (Inverted File with Product Quantization)

A cluster-based ANN index that searches only the few clusters nearest the query and compresses vectors to reduce memory dramatically at some accuracy cost. Reach for it when the dataset is huge and RAM is the binding constraint.

ANN (Approximate Nearest Neighbor)

Index structures used by vector databases that examine only a small fraction of vectors and still return the true nearest neighbors around 99% of the time, trading a sliver of recall for enormous speed. The recall-versus-latency dial is tunable.

LLM-as-Judge

Using a strong model to grade outputs against an explicit rubric. Necessary for open-ended tasks where there is no exact string to match against. The basis of RAGA-style evaluation frameworks for faithfulness and answer relevance.

Red Teaming

Building and running an adversarial evaluation suite — jailbreak attempts, prompt injection payloads, system-prompt extraction, disallowed content requests — on a schedule to quantify your actual security exposure before an attacker does.

Prompt Injection

An attack where malicious instructions are embedded in untrusted content the model reads (web pages, emails, documents). Indirect injection — content addressed to the model in data it retrieves — is the dangerous variant. The model cannot reliably distinguish your instructions from data it just read.

MCP (Model Context Protocol)

An open standard over JSON-RPC that lets any compliant client (your agent) talk to any compliant server (a capability — file system, database, ticketing system) using a stable protocol. Tools are verbs the model can invoke; resources are nouns it can read. Write a server once; every MCP-aware agent can use it.

Skill

A folder containing a skill.md file with YAML front matter (name and trigger description) followed by detailed instructions. The description is the trigger — the model loads the full body on demand via progressive disclosure, keeping context lean while making expertise reusable and versioned.

Cost-Aware Router

An automated layer that estimates the difficulty of each request and routes easy ones to a cheap fast model while sending hard ones to a strong expensive model, capturing most quality at a fraction of the cost. Tuned against your own eval set.

Batch API

An asynchronous API mode for throughput-tolerant workloads (bulk embeddings, overnight processing, eval generation) that processes a large job in the background at roughly half the normal price, with results available within a turnaround window.

OpenTelemetry GenAI Semantic Conventions

An agreed vocabulary for naming spans and attributes in LLM tracing, making model calls appear in the same observability backend as the rest of your services under stable namespace keys rather than proprietary fields.

Clean Core

The SAP enterprise principle that you do not modify the standard system or bury AI calls in frozen code. You expose and consume stable released APIs so that SAP upgrades leave your integrations intact.

Side-by-Side Pattern

The recommended enterprise LLM architecture: build the AI agent as a separate application on the cloud platform (BTP) that reaches into SAP through OData and RAP, keeping the ERP core clean and upgrade-safe.

Memory Tool

A client-side tool the model can call to read and write persistent files in a dedicated memories directory, giving an agent durable cross-session memory. Your code implements the storage backend, access control, and security.

Context Rot

The degradation in model performance as you pack more tokens into a very long context — attention spreads thin and the model gets worse at using any single fact, even one it can technically see. More context is not automatically better.

// FREQUENTLY ASKED QUESTIONS