Frequently Asked Questions About Intellipaat Agentic AI System Design Skill
22 answers covering everything from basics to advanced usage.
// Basics
What exactly counts as generative AI versus a deterministic system?
A system is generative AI only when its output is text, audio, video, or image. If the output is a number, class label, or probability, it is a deterministic system — including decision trees, random forests, XGBoost, CNNs, and RNNs. This distinction matters because the two require different tooling, cost profiles, and evaluation metrics. Never architect a deterministic system as generative AI.
What does 'scalable' actually mean in AI system design?
Scalable means your application works correctly whether 1 million or 50 million customers use it simultaneously, and it does not break under that load. It is a non-negotiable design criterion — every architecture decision must be validated against this before deployment. A solution that works for a demo but collapses at production scale has failed the scalability test.
What is a baseline or foundation model and why don't I build one from scratch?
A baseline model is pre-trained by large organizations on trillions of tokens using expensive GPU clusters like H100s and A100s — costing millions. As an application engineer, you take this model and fine-tune it for your domain or augment it with RAG. Building from scratch is economically irrational; the standard industry workflow is baseline model, then fine-tune or augment, then application.
What are the three steps of a RAG pipeline?
RAG performs Retrieve, Augment, and Generate. Retrieve matches the user query embedding against internal document embeddings using semantic similarity to return relevant context. Augment combines the user query and retrieved context into one single object. Generate passes that augmented input to the LLM, which formats, summarizes, and returns a response. Only add external web search if internal retrieval fails.
Should I be a researcher or an application engineer in AI?
The skill optimizes for becoming an application engineer — someone who applies existing research effectively to solve problems for millions of people at scale. Researchers create something new; application engineers take foundational models and fine-tune, augment, and deploy them to deliver value. For most practitioners building business solutions, the application engineer path is the pragmatic, high-impact choice.
// How To
How do I prepare my internal documents for a RAG system?
Convert whatever format your data is in — PDF, database, code repository, nested structures — into LangChain Document objects with page_content and metadata fields. Metadata must include at minimum the source (WhatsApp, email, phone), type (query, request, complaint), user identity, and timestamp. This Document wrapping is mandatory; the format choice is flexible, but losing metadata breaks access control and auditability.
How do I choose an embedding model for my pipeline?
Use a sentence transformer like sentence-transformers/all-MiniLM-L6-v2, which is free, open-source, and available via HuggingFace. It converts text into numerical vectors. Critically, you must run embeddings on both your internal knowledge base AND incoming user queries — both must live in the same vector space for similarity matching to work. Remember one token is roughly 0.75 English words.
How do I set up access control in a RAG chatbot?
Use metadata to enforce role-based access by filtering retrieved documents before passing context to the LLM. For example, only senior employees can query salary information, or only managers can access certain document classes. This filtering is a software engineering layer on top of vector retrieval — it prevents junior users from extracting restricted information through the chatbot interface.
How do I configure maximum tokens versus context window?
Context window is set by the model provider — the total memory of input plus output tokens the LLM can handle. Maximum tokens is a parameter you set to cap output length per request, for example 512 or 3000 tokens. Always set maximum tokens in production to prevent token waste and control cost. Never confuse the two; exceeding context window causes clipping, while no max token cap wastes money.
// Troubleshooting
My RAG system is hallucinating — what's going wrong?
RAG reduces hallucination by grounding responses in retrieved context, but it does not eliminate it. If the retriever fetches the wrong chunk, the model will hallucinate significantly. Check your embedding quality, chunk sizes, and retrieval relevance. Also verify you aren't exceeding the context window, which causes context rot. RAG improves traceability and reduces hallucination — it is not a guarantee.
Why is my LLM output getting cut off mid-sentence?
Your input plus output tokens are likely exceeding the context window, causing clipping, or your maximum tokens cap is set too low. Check both. If the input is very large, trim or chunk it to stay within the model's context window. If responses are consistently truncated, raise your maximum tokens parameter while staying within the provider's total context window limit.
My HuggingFace embedding model stopped working after an update — why?
HuggingFace libraries change weekly, so a model working today may break tomorrow. Always pin library versions in production rather than using floating versions. If something broke, check which library version changed and revert to a known-good pinned version. This is a common and avoidable production failure in RAG pipelines built on open-source embedding stacks.
My enterprise chatbot gives unpredictable, inconsistent answers — how do I fix it?
Lower your temperature to 0 to 0.3. High temperature increases the model's freedom to pick tokens, producing more creative but less predictable and more hallucination-prone output — wrong for trust-critical enterprise use. Set a low temperature for deterministic, repeatable responses. Also verify your retrieval is returning the correct context chunks, since bad retrieval compounds inconsistency.
// Comparisons
How does a fine-tuned SLM compare to a large foundation model?
A 120 billion parameter model requires 60+ GB RAM and GPU-only inference; a 4 billion parameter model needs just 8 GB RAM and runs on CPU. Fine-tuned small language models are cheaper, edge-compatible, and sufficient for most business tasks. The industry is moving toward SLMs for edge computing. Architect for the smallest model that meets your accuracy requirements.
How does Top-N compare to Top-P for token sampling?
Top-N selects only the top N highest-probability tokens as candidates, improving relevance by discarding low-probability options. Top-P selects tokens whose cumulative probability sums to a threshold like 80%, making it a probability-based alternative that adapts to the distribution shape. Top-P is more flexible when probability is concentrated or spread out, while Top-N uses a fixed count regardless of distribution.
How does FAISS compare to Pinecone or Weaviate?
FAISS is free, in-memory, and runs internal to your code on the same server with no cloud credentials — ideal from small apps through production. Pinecone, Weaviate, and Qdrant are external, cloud-hosted vector databases that offer managed infrastructure, horizontal scaling, and persistence. Choose FAISS for embedded simplicity and cost; choose managed options when you need large-scale, distributed, persistent vector storage.
How does LLM steering compare to fine-tuning?
LLM steering modifies the attention layers to change model behavior and is roughly 50x cheaper than fine-tuning. However, it requires deep technical expertise — incorrect steering produces terrible results. Fine-tuning retrains weights on domain data and is more robust for application engineers without attention-layer expertise. Use steering only if you have the technical depth; otherwise fine-tune or use RAG.
// Advanced
What is model distillation and when should I use it?
Model distillation is a training strategy where multiple large teacher models teach a smaller student model, letting the student achieve strong performance without training from scratch on massive data. Use it when your budget is very tight and you need a compact, deployable model — particularly for edge deployment of fine-tuned small language models where full fine-tuning on huge datasets is impractical.
When does RAG fail at scale and what should I do instead?
RAG struggles at scale with millions of documents, suffers retrieval failures that cause large hallucinations, and has faithfulness versus correctness conflicts. At massive scale, consider hybrid approaches, better chunking and metadata filtering, re-ranking layers, or escalating to agentic patterns that decompose tasks. Do not force RAG everywhere — use it where it fits and architect around its known failure modes.
What is context rot and how do I prevent it?
Context rot, also called context poisoning, is the degradation of LLM output quality when the context window is exceeded or irrelevant context accumulates, causing the model to lose track of earlier information. Prevent it by staying within context window limits, keeping prompts concise, filtering retrieved chunks for relevance, and avoiding dumping excessive or off-topic content into the input.
When should I add external web search to my RAG pipeline?
Only add external search like Tavily when internal retrieval fails to answer the query — never default to it. Internal RAG should be your first resort because it is grounded in your proprietary, access-controlled knowledge. Defaulting to external web search before exhausting internal retrieval introduces untrusted information, breaks traceability, and undermines the whole point of a private knowledge base.
What carbon and cost factors should I account for before deploying an LLM?
LLMs carry real energy, GPU, and inference costs plus a significant carbon footprint — a 220 million parameter model can generate carbon equivalent to 250,000 people per year. Factor in these externalities alongside infrastructure budget and scalability. This is central to cost-first architecture: always prefer a cheaper alternative if it does the task, and justify LLM use against its full cost profile.