Intellipaat Agentic AI System Design Skill

// When should you use the Intellipaat Agentic AI System Design skill?

Use this skill whenever you need to architect an AI solution for a business problem, decide which AI technology stack to deploy (classical ML vs LLM vs RAG vs agents), configure LLM inference parameters, or build a retrieval-augmented generation pipeline from scratch.

// What do you need before designing an agentic AI system?

• Business Problem Statementrequired
  What task or workflow the AI system must solve, including who the end users are and what decisions the system must make or support.
• Data Descriptionrequired
  What internal or external data is available — format (PDF, DB, code, text), volume, and access controls (who can query what).
• Cost and Scale Constraintsrequired
  Approximate user load (scalability target), infrastructure budget, and whether cloud or on-premise/edge deployment is required.
• Accuracy vs. Creativity Requirements
  Whether the system needs deterministic/standard responses (e.g. enterprise chatbot) or creative/exploratory outputs (e.g. content generation).
• Existing Codebase or Knowledge Base
  Any internal documents, code repositories, or structured data that will serve as the RAG knowledge source.

// What principles guide cost-first agentic AI architecture?

// How do you design an agentic AI system step by step?

1. 1

   Classify the problem type before selecting any AI technology

   Ask: Is the output a number, class label, or probability? → Use classical ML (decision tree, XGBoost, CNN, RNN). Is the output text, image, audio, or video? → Consider Generative AI. Is the task repetitive and decision-based with cheap data? → Use classical ML. Only escalate to LLMs when classical approaches are insufficient or when the generative output type is required.

2. 2

   Evaluate cost and scalability constraints against technology options

   LLMs are not cheap to build, deploy, or use — account for energy, GPU cost, inference cost, and carbon footprint (a 220 million parameter model generates carbon equivalent to 250,000 people per year). Always prefer the cheaper alternative if it does the task. Validate that your chosen architecture is scalable (works at 1M and 50M users without breaking).

3. 3

   Select the model tier: Foundational/Baseline Model, Fine-Tuned SLM, or RAG-augmented LLM

   Baseline/Foundation Models: pre-trained on trillions of tokens, built with billions of parameters on expensive GPU clusters (H100s, A100s). Use as-is for general tasks. Fine-Tuned SLM: take a baseline model and fine-tune it on your domain-specific data — preferred for edge deployment, cost reduction, and task specificity. RAG-augmented LLM: keep baseline model frozen; augment with internal knowledge via vector retrieval. Use when data is proprietary, frequently updated, or behind access controls.

4. 4

   Prepare internal documents as LangChain Document objects with metadata

   Whatever format your internal data is in (PDF, database, code repository, list, nested structure) — convert it to LangChain Document objects with page_content and metadata fields. Metadata must include at minimum: source (WhatsApp/email/phone), type (query/request/complaint), user identity, and timestamp. This is the software engineer's job — format choice is flexible but Document wrapping is mandatory for RAG.

5. 5

   Convert Document objects to embeddings using a sentence transformer embedding model

   Use an embedding model such as sentence-transformers/all-MiniLM-L6-v2 (free, open-source, available via HuggingFace). This model converts text into numerical vector representations. One token ≈ 0.75 English words. Run embeddings on both the internal knowledge base AND on incoming user queries — both must be in the same vector space for similarity matching to work.

6. 6

   Store embeddings in a Vector DB using FAISS (or production-grade alternative)

   FAISS (built by Meta/Facebook) is a free, in-memory vector DB for efficient similarity search and clustering of dense vectors — suitable for small to production-scale applications. For external/cloud-hosted vector storage, alternatives include Pinecone, Weaviate, or Qdrant. FAISS runs internal to the code on the same server — no separate cloud credentials required. Use from_documents() method passing both docs and embedding model.

7. 7

   Implement the RAG three-step pipeline: Retrieve → Augment → Generate

   Retrieve: match user query embedding against internal document embeddings using semantic similarity — return the most relevant context chunks. Augment: combine the user query and the retrieved context into one single object (augmentation stage) — this combined object is the LLM's input. Generate: pass augmented input to the LLM to format, summarize, and return a response. Only add external search (e.g. Tavily web search) if internal retrieval fails to answer — do not default to external search.

8. 8

   Configure LLM inference parameters: Context Window, Maximum Tokens, Temperature, Top-P, Top-N

   Context Window: set by the model provider — the total memory (input + output tokens combined) the LLM can handle. Never exceed it or output gets clipped. Maximum Tokens: YOU set this — caps the output token length per request (e.g. 512 or 3000 tokens). Always set this in production to prevent token waste. Temperature: 0–0.3 for enterprise/standard responses; 0.7–1.0 for balanced creativity; 1.5–2.0 for creative/generative tasks. Higher temperature = more hallucination risk. Top-N: select only the top N highest-probability tokens as candidates — improves relevance. Top-P: select tokens whose cumulative probability adds up to P% (e.g. 80%) — a probability-based alternative to Top-N.

9. 9

   Apply access controls and metadata filters to RAG retrieval

   Use metadata to enforce role-based access: e.g. only senior employees can query salary information, only managers can access certain document classes. This is a software engineering layer on top of the vector retrieval — filter by metadata before returning context to the LLM. This prevents junior users from accessing restricted information through the chatbot.

10. 10

    Evaluate whether to escalate from RAG to Agents or Multi-Agent Systems

    RAG is appropriate for internal knowledge retrieval at moderate scale. Escalate to AI Agents when the system must execute real business workflows, take decisions autonomously, coordinate across multiple tools or systems, or complete end-to-end tasks with minimal human input. Multi-agent frameworks (CrewAI, AutoGen, LangGraph) are appropriate when tasks require coordination between specialized agents. Agentic AI and AI Agents are not the same thing — clarify which pattern the business problem requires.

// What do real agentic AI system designs look like in practice?

// What mistakes should you avoid when building agentic AI systems?

• Reaching for an LLM when a classical ML model (decision tree, XGBoost, CNN, RNN) would be cheaper, faster, and equally accurate for the task — always do cost-first technology selection.
• Calling a system 'Generative AI' when the output is a number, class label, or probability — these are deterministic systems and require different architecture and evaluation.
• Treating Agentic AI and AI Agents as the same thing — they are distinct concepts with different architectural implications.
• Converting raw text directly to embeddings without wrapping in a Document object — you lose metadata (source, type, user, timestamp) which is essential for access control, audit, and richer retrieval.
• Using RAG as the answer to every knowledge problem — RAG fails at scale (millions of documents), suffers retrieval failures that cause large hallucinations, and has faithfulness vs. correctness conflicts.
• Defaulting to external search (e.g. web search) before exhausting internal retrieval — only add external search if internal RAG cannot answer the query.
• Setting no Maximum Tokens limit in production — without a cap, LLM APIs waste tokens and cost money; always set a maximum token guardrail.
• Using a high temperature for enterprise or customer-facing chatbots — high temperature increases hallucinations and unpredictability; use low temperature (0–0.3) for standard, trust-critical responses.
• Ignoring context window limits and passing very large inputs — input + output tokens exceeding the context window causes clipping, incomplete outputs, and context rot/context poisoning.
• Fine-tuning or steering LLMs without sufficient domain knowledge — LLM steering (modifying attention layers) is 50x cheaper than fine-tuning but requires deep technical expertise; incorrect steering produces terrible results.
• Assuming RAG eliminates hallucination — RAG reduces hallucinations by grounding responses in retrieved context, but if the retriever fetches the wrong chunk, the model will hallucinate significantly. RAG improves traceability and reduces hallucination; it does not eliminate it.
• Using HuggingFace embedding libraries without version-pinning — HuggingFace libraries change weekly; a model working today may not work tomorrow. Always pin library versions in production.

// What key terms should you know for agentic AI system design?

Agentic AI

AI that behaves less like a tool and more like a digital workforce — it acts, decides, and completes end-to-end tasks autonomously with minimal human input. Distinct from AI Agents.

AI Agents

Specific autonomous software entities that perceive inputs, make decisions, and take actions using tools. Distinct from Agentic AI as a broader paradigm.

Baseline Model / Foundation Model / Pre-trained Model

A model trained by large organizations on massive unstructured data using expensive GPU clusters. The starting point that application engineers fine-tune or augment — not built from scratch.

Generative AI

An AI system whose output is text, audio, video, or image. If output is a number, class label, or probability, it is a deterministic system — not Generative AI.

Deterministic System

An AI system with clear determinism in its mathematics — outputs a probability, label, or number. Includes decision trees, random forest, XGBoost, CNNs, RNNs. Not Generative AI.

RAG (Retrieval Augmented Generation)

A three-step pipeline: Retrieve relevant context from a vector DB, Augment (combine query + context into one object), Generate a response via LLM. Ideal for internal enterprise knowledge; not a silver bullet at scale.

Embedding

Converting text (a word, sentence, or document) into a numerical vector representation. Example: the word 'apple' becomes [0.1, 0.69, 0.3, 0.2]. Required before storing text in a Vector DB.

Vector DB

A database that stores embeddings (vectors) and supports efficient similarity search. Examples: FAISS (free, in-memory, internal to code), Pinecone, Weaviate, Qdrant.

FAISS

A library built by Meta/Facebook for efficient similarity search and clustering of dense vectors. A free, in-memory Vector DB that runs internal to the code — suitable from small applications through production.

Augmentation Stage

The second step of RAG — combining the user's query embedding and the retrieved internal context into one single object before passing to the LLM for generation.

Document Object

LangChain's internal data structure for text. Contains page_content (the text) and metadata (source, type, user, timestamp, etc.). Mandatory wrapper before converting text to embeddings in a RAG pipeline.

Context Window

The memory or attention span of a large language model — the maximum number of tokens (input + output combined) the model can process at once. Set by the model provider. Exceeding it causes clipping and context rot.

Maximum Tokens

A parameter YOU set to cap the number of output tokens the LLM generates per request. Distinct from context window (which is set by the model provider). Essential for cost control in production.

Context Rot / Context Poisoning

Degradation of LLM output quality that occurs when the context window is exceeded or when irrelevant context accumulates, causing the model to lose track of earlier information.

Temperature

A distributional philosophy parameter controlling how much freedom the LLM has when selecting output tokens. Lower temperature = less freedom, more confident, more repetitive, fewer hallucinations. Higher temperature = more freedom, more creative, more hallucinations.

Top-N

An inference parameter that selects only the top N highest-probability tokens as candidates for the model's output. Improves relevance by discarding low-probability tokens.

Top-P

An inference parameter that selects tokens whose cumulative probability sums to up to P% (e.g. 80%). A probability-based alternative to Top-N for controlling output token diversity.

Fine-Tuned SLM (Small Language Model)

A small language model (e.g. 3-4 billion parameters) that has been fine-tuned on domain-specific data. The predicted future of AI deployment — cheaper, edge-compatible, and sufficient for most business tasks.

Model Distillation / Mixture of Experts

A training strategy where multiple large teacher models teach a smaller student model, allowing the student to achieve strong performance without training from scratch on massive data.

LLM Steering

Modifying the attention layers of an LLM to change its behavior without full fine-tuning. 50x cheaper than fine-tuning but requires deep technical expertise — incorrect steering produces terrible results.

Transformers / Attention Is All You Need

The 2017 Google Brain paper by Ashish Vaswani et al. that introduced multi-headed self-attention, replacing sequential RNN processing with parallel token processing and fundamentally changing the AI landscape.

Inference

The process by which a trained LLM predicts the next token and generates a completion given an input prompt. Inference configuration parameters (temperature, max tokens, top-P, top-N) control this process.

BPE (Byte Pair Encoding)

The subword tokenization method used by transformer models. Converts text into tokens (subword units) before embedding. One token ≈ 0.75 English words.

HuggingFace

A platform where every major AI model (Google, Meta, etc.) is uploaded and made accessible. Allows running, downloading, and fine-tuning models. Note: libraries update frequently — pin versions in production.

Scalable

A system is scalable if it works correctly and without breaking whether 1 million or 50 million customers use it simultaneously. A non-negotiable design criterion.

// FREQUENTLY ASKED QUESTIONS