Beyond the buzzwords: how to build a RAG chatbot?

What is RAG?

RAG stands for Retrieval-Augmented Generation which is a hybrid AI approach that combines two fundamental processes: retrieval and generation. Unlike traditional large language models that depend solely on their training data (which becomes outdated over time), RAG systems dynamically pull relevant information from current knowledge bases like documents, databases, or specialized repositories before generating responses.

This architecture addresses a critical business challenge: maintaining accurate, up-to-date information delivery. RAG enables organizations to leverage their existing documentation, procedures and knowledge bases.

The process works through three core steps: Retrieve (finding relevant context from your knowledge base), Augment (combining the user’s query with retrieved information) and Generate (creating contextually appropriate responses using a large language model).

In practice, this means when a user asks about a specific procedure, the system embeds their query as a vector representation, searches through your document database using similarity matching (typically cosine similarity or semantic search), retrieves the most relevant information and then provides a comprehensive answer grounded in your actual business content rather than potentially outdated AI training data.

RAG Chatbot Architecture

Search

The search phase forms the foundation of effective RAG systems. When users submit questions, the system must efficiently locate relevant information from potentially vast document collections through two critical processes: building a searchable index and executing precise retrieval operations.

The first step is index construction. Documents must be transformed into a structured, searchable format. This involves segmenting content into coherent chunks, typically between 300 and 500 tokens, with overlapping regions to preserve semantic continuity. Poor chunking decisions often degrade retrieval precision. Each chunk is converted into a vector representation through an embedding model. These vectors are stored alongside metadata such as source, document type, timestamps and semantic labels. Metadata enrichment is not optional; it enables filtered search, improves ranking and supports explainability.

Advanced preprocessing tools, such as Azure Document Intelligence, can preserve structural elements like headings, tables and sections. Maintaining structural hierarchy often improves semantic retrieval performance.

As mentioned before the chunks are translated into vector representations through an embedding model. Some examples in Azure are given below, please note that the landscape changes rapidly over time, so it is always best practice to investigate the state-of-the-art models.

• Azure OpenAI Embeddings:
  • text-embedding-ada-002: Cost-efficient and fast (little bit outdated)
  • text-embedding-3-small: Optimal cost-performance balance
  • text-embedding-3-large: Highest accuracy

Models such as text-embedding-3-small provide cost-efficient performance, whereas larger embedding models offer improved semantic resolution at higher cost. The choice should be made based on retrieval benchmarks.

The next step is the search strategy. The system can use diverse search strategies: vector search (semantic similarity in embedding space), semantic search (understanding deeper meaning and intent), hybrid search (combining vector similarity with keyword matching) and custom search incorporating metadata fields and specialized logic.

Optimization involves tuning parameters like ef_search and ef_construction (controlling speed-accuracy trade-offs), top-k selection (balancing coverage with precision) and relevancy thresholds (filtering irrelevant results without being overly restrictive). To summarize, retrieval is not static, but requires iterative calibration against evaluation metrics.

Augment & Generate

To improve the answers of the chatbot, the augmentation phase bridges retrieval and generation by combining user queries with retrieved context to create enriched prompts for the language model. This step significantly impacts response quality and benefits from several enhancement techniques.

There are different layers where improvements can be added:

• Global constraint – instructions
• Pre-Retrieval Optimization – Improving the query before searching.
• Post-Retrieval Optimization – Improving document selection after searching.
• Generation – Producing the final answer under defined constraints.

Instruction Prompt

This is a context augmentation technique where we define instructions to the system. This gets added before the prompt of the end user and lets you steer the behavior of the model based on the specific needs and use cases. It also persists throughout the interactions with the chatbot. The reason for an instruction prompt is to reduce the risk of hallucinations and have robust answers in the way the user prefers.

An instruction prompt could have many components:

• Defining a persona or role (“You are a Senior Data Analyst”)
• Defining an output format (“Answer in Markdown structure”)
• Defining goals or rules (Return code comments at each line)
• Providing additional context (“The provided context is about Intellus”)
• Defining constraints (“You must state information not available if the answer is not provided in the context”)

An example of an instruction prompt:

Pre-Retieval Optimization

Pre-retrieval techniques improve the query before it is embedded and sent to the vector database. Their purpose is to increase recall and reduce ambiguity. This will improve the results that are being retrieved from the vector database.

Conversation history & Contexual Rewriting

Chatbots operate in multi-turn conversations. Users frequently refer to earlier outputs using implicit language such as “it,” “that model,” or “the last part.” If such queries are embedded directly, retrieval quality decreases because the semantic context is incomplete.

Contextual rewriting resolves this issue.

The system uses conversation history and the new input to generate a standalone query. Importantly, the model is instructed only to rewrite the query, not to answer it.

Example of a contextual rewrite prompt:

Query Expansion

Standard retrievals usually suffer from mismatches due to terms used by the user that are not matching with the terminology present in the context. To reduce this problem, RAG systems utilize Query Expansion. This process uses an LLM to generate multiple variations of the original query or hypothetical content before retrieval, effectively broadening the probability of finding relevant context.

There are certain techniques discussed with examples:

Multi-Query Expansion (paraphrasing):
This technique makes use of an LLM to rewrite the input from the user into multiple variations using different perspectives, reformulation or even synonyms.

HyDE (Hypothetical Document Embeddings)
HyDE generates hypothetical answers to the input of the user and uses its embeddings to refine the search in the context such that the answers are more semantically aligned.

Related Question Expansion:
This approach broadens the search context by generating questions that are semantically similar to the user’s input.

Post-Retrieval Optimization

Post-retrieval techniques refine the set of retrieved documents before generation. Their purpose is to increase precision.

Reranking

Standard vector searches retrieve a large set of candidates. However, this can find everything that might be related to precise answers. Finding documents that could share keywords or general topics with the user’s question but don’t actually contain the specific answer.

Reranking is used to refine and optimize the relevance of the retrieved context before the generation phase. The first retrieval will return a set of candidate matches with the input and reranking will reorder these candidates with multiple approaches:

Cross-Encoder Reranking:
This method scores (through a transformer model) the user’s input against the retrieved candidates. This kind of method could be slow and computationally heavy as there could be many candidates to check individually against the input.

LLM-Based Reranking:
This approach  uses an LLM to rank the candidates from most useful to least useful. The documents could be presented to the LLM with an instruction as such: “You are an expert evaluator. Rank these passages based on how well they answer the user’s question. If a passage is irrelevant, discard it.“

The downside of this method that it could be costly because we are not making use of a model like in Cross-Encoder Reranking, but we are making use of an LLM which consumes tokens.

When are rerankers most useful?
These are some situations where it might be useful:

• Long, Complex Boolean Queries

• Adjacent-Concept Queries

Generation

The final stage of the RAG pipeline is generation after retrieval and reranking the most relevant information. The goal is to create a response to satisfy the user.

Conversation Window

While rewriting optimizes the search, windowing optimizes the generation. Before each response, the chatbot can also make use of the recent conversation turns and use it as context. The context limit can be defined to make use of X amount of turns, so only the most recent X inputs + outputs can be included while older turns are discarded. The reason for this limit can be due to token or budget limits. In practice, it will work by storing each message in a buffer. When a new input has been added by the user then the last X turns will be selected as context. This will be included when generating a response.

Example of sliding window:

Model Selection & Hyperparameter Tuning

The choice of the appropriate LLM involves a trade-off between reasoning capabilities, latency and cost.

• High-Reasoning Models: These models are able to process advanced logic and can follow instructions. They are used for complex tasks to reason, analyze or even generate code. The downside is that it can be costly.
• High-Efficiency Models: These models are optimized for speed and cost-effectiveness. They are ideal for simple retrieval tasks when the answer is explicitly defined in the text, so deep reasoning is not required.

The differences in the models and their pricing can be compared on Azure.

The behavior of the generation is also controlled by hyperparameter tuning such as temperature, which controls the randomness (creativity vs. predictability) of an LLM.

• Low Temperature (0 – 0.3): The result will be more deterministic and best suited for use cases where the response should be more consistent
• High Temperature: (0.7-1.0): Best suited for creative outputs like brainstorming or get more understanding of technological concepts with real life examples.

Examples:

Evaluation

Comprehensive evaluation ensures chatbot reliability and provides metrics for continuous improvement. Organizations should implement both automated assessment and human evaluation to maintain high performance standards and demonstrate business value.

The evaluation process involves testing different RAG configurations by creating question-answer pairs from the knowledge base, running queries through various setups and measuring performance against expected results. This systematic approach enables data-driven optimization of system parameters.

Retrieval Evaluation

Retrieval must be evaluated independently from generation. If the wrong documents are retrieved, even a strong language model cannot produce a correct answer.

One core metric is Mean Average Precision (MAP). MAP measures not only whether the correct document was retrieved, but also how high it appears in the ranked results. If the relevant document appears near the top, the score increases. If it is buried at lower ranks, the score decreases. A low MAP score indicates that ranking or embedding strategies require adjustment.

To assess similarity between retrieved chunks and ground truth passages, tools such as spaCy can be used to compute textual similarity. If retrieval consistently returns irrelevant content while generation appears fluent, the issue lies in the retrieval configuration rather than the model.

Retrieval evaluation primarily measures:

• Ranking quality
• Recall of relevant documents
• Precision of top-k results

These metrics directly reflect the effectiveness of pre-retrieval and post-retrieval optimization techniques discussed earlier.

Generation Evaluation

Generation quality is assessed by comparing model outputs with ground truth answers. This is typically performed using end-to-end metrics.

Distance-based metrics measure textual similarity between generated and expected answers.

Cosine similarity measures vector alignment in embedding space. Variants such as cosine and cosine Ochiai indicate how closely two answers align semantically.

Semantic similarity models provide deeper meaning comparison using pretrained embeddings. These approaches measure whether the generated response conveys the same meaning as the ground truth, even if wording differs.

LLM-based evaluation provides qualitative assessment at scale by using language models as judges. These metrics evaluate whether retrieved context helps answer questions, assess answer relevance to user queries and measure contextual precision and recall. While not fully deterministic, they offer practical quality assessment without extensive manual review.

Business stakeholders should focus on metrics that correlate with operational improvements: response accuracy rates, user satisfaction scores, reduction in support ticket volumes and time-to-resolution for customer queries. These measurements directly translate technical performance into business value.

Buy vs Build

When implementing a chatbot with RAG. Organizations have multiple options to go for. They must determine whether to adapt an existing platform, build a custom RAG pipeline or use low-code or no-code frameworks. This will be discussed in the upcoming sections.

Build: Custom Rag Development

The build path means creating your own RAG pipeline from the ground up using a certain cloud AI infrastructure. This is the approach we advise to take when the organization wants full control over every architectural decision. For example, how documents are chunked, which embedding model is used, how retrieval is tuned and how the system is tied all together.

In practice, every major cloud provider offers services like this. On Microsoft Azure, Azure AI Foundry provides a managed environment for model deployments, vector indexing and pipeline orchestration. On AWS, Amazon Bedrock offers the equivalent capabilities. On Google Cloud, Vertex AI serves a similar role as well. The specific platform matters less than the principle: you need an environment which gives the leverage to use LLM APIs, host a vector database and allow you to wire together the retrieval and generation stages described in Section 2.

The build approach unlocks the full optimizations discussed in the previous sections like hybrid search, reranking, query expansion, custom chunking strategies and evaluation. It also integrates cleanly with existing organization data infrastructure.

The biggest trade-off that comes with build is the investment. A custom pipeline requires engineering capacity and ongoing maintenance. It would be the right choice if retrieval quality and response accuracy are business-critical and should be completely in the hands of the organization.

Practical Example: Webinar Video Data

To illustrate what a custom build delivers in practice, consider a solution we developed that transforms a library of recorded webinar content into a queryable knowledge base. Rather than requiring users to watch hours of video, they can ask direct questions and receive precise answers drawn from the transcripts.

The pipeline works in two stages. During ingestion, video content is transcribed using OpenAI’s Whisper model, segmented into semantically coherent chunks and indexed as vector embeddings. During retrieval and generation, incoming user queries are matched against the index, the most relevant transcript segments are retrieved and a language model synthesizes a natural, context-aware response. Conversation history is maintained across turns, so follow-up questions work seamlessly without the user needing to repeat context.

This architecture generalizes to other scenarios such as:

• A customer support assistant
• A legal document analysis tool for reviewing contracts or regulatory materials
• An internal AI assistant for employees interacting with company knowledge bases

Low-Code

For organizations that need working chatbot functionality quickly without a dedicated engineering team then low-code platforms are a good middleground for this. Microsoft Copilot Studio is one of the most widely adopted options in this space and especially if the organization is already embedding in Microsoft’s ecosystem.

Copilot Studio offers a graphical, easy to use interface for building chatbots. It connects out of the box with the supported knowledge sources:

The limitations that come together with this approach is that the advanced techniques discussed in Section 2 are not easy to adopt or even unavailable. Our recommendation is to treat low-code platforms as a validated starting point. They are good for testing whether your data sources are structured well enough to support a RAG scenario and for delivering business value while a more custom solution is defined.

Buy

The last option is to purchase a fully prebuilt chatbot. This is a SaaS solution where the vendor already has built the RAG infrastructure, the interface and the needed optimizations. The organization only has to provide the data sources and the building and managing part are done by the vendor.

This is more appealing when minimal technical efforts are needed. The constraints that come with this solution is that the retrieval quality, model choice, chunking strategy and evaluation methodology are all black boxes. When the responses are not really in line with the expectations then the organization is only limited to the available configuration by the vendor.

This path is best suited to organizations with a well-defined, stable use case that maps closely to what the vendor’s product was designed to handle and where deep customization of the AI behavior is not a priority.

Choosing the right path

The decision taken by the organization is rarely permanent. Some may start with a low-code or a fully prebuilt chatbot to see its value directly. After that organizations could migrate toward a custom build as requirements would grow. What matters most is that the initial choice is driven by the expectations of the organization.

Conclusion

RAG represents a significant step forward in making AI systems practical for real-world business use. By grounding language model responses in an organization’s own up-to-date knowledge, it addresses the core limitations of static training data and reduces the risk of hallucinations.

The quality of a RAG system is determined at every stage of the pipeline. Effective retrieval depends on thoughtful chunking, the right embedding model and a search strategy tuned to the use case. Augmentation techniques for example contextual query rewriting and query expansion to reranking will ensure that the most relevant information reaches the generation stage. And model selection, temperature tuning and conversation windowing shape the final response into something accurate, consistent and useful.

Evaluation cannot be an afterthought. Measuring both retrieval precision and generation quality through automated metrics and human review provides the feedback loop necessary for continuous improvement. Without it, degradation in system performance can go unnoticed until it impacts users.

On the question of build versus buy, there is no universally correct answer. Low-code platforms offer a fast path to validating a use case, prebuilt SaaS solutions minimize technical overhead for well-defined scenarios and custom pipelines deliver the control and optimization depth that business-critical applications demand. The right starting point depends on the organization’s maturity, priorities and available resources.

Ultimately, RAG is not a product to deploy once and forget. It is an architecture that needs ongoing investment in data quality, retrieval tuning and evaluation discipline. Organizations that treat it as a living system rather than a one-time implementation will consistently see the greatest return.

---