How to Build a Chat-with-PDF SaaS Using RAG
For Indie SaaS developers building chat-with-PDF products · Based on Sujan Anand RAG Application Build Framework
// TL;DR
This guide helps indie SaaS developers build a chat-with-any-PDF product using the RAG Application Build Framework. Users upload a PDF through a Streamlit frontend that POSTs to a FastAPI /ingest endpoint, which chunks, embeds, and stores the document in ChromaDB. The /ask endpoint runs semantic search and returns a grounded LLM answer plus the retrieved chunk sources. Chat history persists in st.session_state across reruns. The full three-part architecture — frontend, backend, vector store — gives you a shippable product where every answer cites its source page and unknown questions return 'I do not know.'
Why is RAG the right architecture for a chat-with-PDF product?
Users uploading arbitrary PDFs expect accurate, cited answers — not confident hallucinations. The RAG Application Build Framework delivers exactly that with a clean three-part architecture: a Streamlit frontend for upload and chat, a FastAPI backend that ingests and answers, and ChromaDB as the vector store. Because retrieval happens at query time, your product answers from whatever document the user just uploaded, cites the exact page, and stays cheap by injecting only the top few relevant chunks per question instead of the whole file.
How do you wire up the upload-and-ingest flow?
Build a Streamlit sidebar with st.file_uploader(type=['pdf']) and an 'Ingest PDF' button that POSTs the file to your FastAPI /ingest endpoint with a spinner. On the backend, /ingest accepts an UploadFile, saves it to a temp file with tempfile.NamedTemporaryFile(delete=False), runs the full Offline Pipeline — extract with PyMuPDF, chunk at 500 characters with 50-character overlap, batch-embed 100 chunks per call, store in ChromaDB with page metadata — then deletes the temp file with os.unlink() and returns chunks_added and total_chunks. Show that chunk count in the UI so users know ingestion worked.
How do you handle the chat and retrieval flow?
Use st.chat_input with the walrus operator to capture questions, and store every message in st.session_state['messages'] so the conversation survives Streamlit's reruns. Each question POSTs to /ask, which accepts a Pydantic AskRequest(question, n_results=3), embeds the question with the same model as the chunks, runs collection.query for the nearest chunks, injects them into the context-only prompt, calls gpt-4o-mini, and returns the answer plus sources. Render each answer with an st.expander showing chunk text, page metadata, and similarity score — this explainability is a genuine product differentiator.
How do you make it production-ready?
Enable CORS middleware on FastAPI so your frontend can reach the backend, and add a GET / health-check returning status and total_chunks. Verify everything at localhost:8000/docs, where FastAPI auto-generates interactive API documentation for both endpoints. For scale, swap ChromaDB for a managed vector database like Pinecone or pgvector, add per-user collection isolation so one customer's documents never leak into another's answers, and implement incremental ingestion to avoid re-embedding unchanged files.
What pitfalls kill chat-with-PDF products?
Dumping an entire PDF as one embedding is the classic mistake — embedding models have token limits and you can't retrieve sub-sections from a monolithic vector, so always chunk first. Forgetting overlap splits sentences at boundaries and produces nonsense chunks. Skipping the context-only prompt lets the model hallucinate, destroying user trust. And re-ingesting without clearing the collection creates duplicate chunks. Handle each of these and your product answers reliably.
Next step
Stand up the two-terminal dev loop: run uvicorn backend:app --reload, then streamlit run frontend.py. Upload a test PDF, confirm the chunk count appears, ask a few questions, and verify answers cite pages while off-topic questions return 'I do not know.' Once that loop is solid, add per-user isolation and swap in a production vector database.
// FREQUENTLY ASKED QUESTIONS
How do I keep one user's PDFs from leaking into another user's answers?
Isolate documents per user by using separate ChromaDB collections or metadata filters keyed to a user ID, and scope every /ask query to that user's collection. This ensures semantic search only retrieves chunks belonging to the requesting user, preventing cross-tenant data leakage in a multi-user SaaS.
How do I let users chat with a PDF right after uploading it?
Wire the Streamlit uploader to POST the file to your FastAPI /ingest endpoint, which runs the full Offline Pipeline and stores chunks in ChromaDB. Once ingestion returns a chunk count, the /ask endpoint can immediately run semantic search against those chunks, so users can start chatting the moment ingestion completes.
How do I keep costs low as usage grows?
Batch 100 chunks per embedding call during ingestion, retrieve only the top 3 chunks per query, and use gpt-4o-mini for generation. Since RAG injects only relevant chunks rather than whole documents, per-query token cost stays small. Cache or skip re-embedding unchanged documents to avoid redundant embedding spend.