Getting appropriate responses from a RAG empowered LLM is an art. The primary 3 knobs to adjust are the RAG System Prompt, Document Return Count, and Chunk Size.

Use this guide to help you fine tune your RAG empowered LLM responses.

RAG System Prompt

To edit your RAG system prompt, navigate to Admin Panel→RAG Config

A system prompt is a set of instructions that defines how an AI assistant should behave. In a RAG (Retrieval-Augmented Generation) model, the system prompt is important because it tells the assistant how to use your uploaded documents, what to treat as the “source of truth,” and how to respond when the documents don’t contain an answer.

A strong RAG system prompt should include a few key elements:

1. The assistant’s role (what kind of helper it is)
2. Its main goal (what it’s trying to accomplish)
3. How it should use the documents
4. What it should do when information is missing. If hallucination prevention matters (and it usually does), you should explicitly tell the assistant not to guess or fill in gaps, and instead to say when the documents don’t provide enough information.

Here is a generic example system prompt:

“You are a document-based AI assistant. Your job is to answer user questions using the provided documents as the primary source of truth. If the answer is clearly stated in the documents, respond directly and accurately. If the documents do not contain enough information, say: ‘The provided documents do not contain sufficient information to answer this question.’ Do not guess or fabricate information. Keep responses clear, concise, and factual.”

Additional Considerations for Your RAG System Prompt

• Document Strictness: How strictly do you want the assistant to rely only on your uploaded documents versus using its general knowledge?
• Evidence Requirement: Do you want the assistant to cite sources or include direct quotes from your documents to support its answers?
• Missing Information Policy: What should the assistant do if the answer cannot be found in your documents — say “I don’t know,” ask for clarification, or provide a best-effort response?
• Inference Boundaries: Should the assistant only repeat what is explicitly stated in the documents, or can it draw reasonable conclusions from them?
• Uncertainty Transparency: Should the assistant clearly indicate when information is incomplete, ambiguous, or conflicting in the documents?
• Response Style: How would you like the assistant to structure its responses — concise, detailed, step-by-step, or in bullet points?