Make an AI answer from your own documents: RAG explained and self-hosted

By Mouhamed BANKOLE, CEO at ITNET Technologies · Updated November 6, 2026 · Reading time: 9 min

A general-purpose AI assistant knows nothing about your procedures, your contracts or your product documentation, and it sometimes makes up a plausible answer. RAG AI fixes that: before answering, the AI looks up the relevant passages in your own documents and relies on them. It is now the simplest way to build a reliable internal assistant. Here is how RAG works, what it is used for in a small business, how it differs from fine-tuning, and how to host it yourself on a GPU server.

In short

• RAG (retrieval-augmented generation) makes the AI search your documents for relevant passages before it answers.
• Four building blocks: document chunking, embeddings, a vector database, then a language model that writes the answer from the passages found.
• It does not change the model: you add or remove a document without retraining anything, unlike fine-tuning.
• Hosting it yourself keeps sensitive documents in-house, on a server in France.
• A GPU server runs the model; Ollama, a vector database and Open WebUI are enough to get started.

Contents

1. What is RAG in AI?
2. How does RAG work, step by step?
3. What is RAG used for in a small business?
4. RAG or fine-tuning: what is the difference?
5. Why host your RAG yourself?
6. What architecture for a self-hosted RAG?
7. What server do you need for RAG?
8. Which mistakes should you avoid with RAG?
9. How do you get started with RAG quickly?

What is RAG in AI?

RAG is a method that combines a search engine over your documents with a language model: the AI first retrieves the relevant passages, then writes its answer from them. The term comes from the research paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" published by Facebook AI researchers.

What RAG changes for the user:

• grounded answers based on your documents rather than the model's general memory;
• citable sources: the assistant can show the document and passage it used;
• up-to-date knowledge: you only need to add the new document;
• fewer made-up answers: the model answers from text it was given.

How does RAG work, step by step?

A RAG system first prepares your documents as indexed chunks, then, for each question, retrieves the closest chunks and passes them to the model along with the question. It happens in two phases: indexing, once, and answering, on every request.

1. Chunking: each document is split into passages of a few paragraphs.
2. Embeddings: a specialized model turns each passage into a vector, a list of numbers that represents its meaning.
3. Vector database: the vectors are stored in a database that can find the nearest ones.
4. Search: the question is also turned into a vector, and the database returns the closest passages.
5. Generation: the language model receives the question and those passages, and writes the answer.

An embedding model such as nomic-embed-text weighs only 274 MB and is called through the Ollama API: the "search" part does not need a big machine.

What is RAG used for in a small business?

RAG lets you query a mass of internal documents in plain language: procedures, technical documentation, contracts, the support knowledge base. It replaces keyword searches in shared folders.

• RAG does not decide: it helps find and rephrase; validation stays human.
• Document quality matters: an outdated document gives an outdated answer.

RAG or fine-tuning: what is the difference?

RAG gives documents to the model at question time, while fine-tuning changes the model itself through new training: to make an AI know your documents, RAG is almost always the right choice.

• Start with RAG, and keep fine-tuning for cases where the model must adopt a particular style or format.

Why host your RAG yourself?

Because RAG by definition handles your internal documents: hosting it yourself avoids sending them to an AI vendor and simplifies GDPR compliance. It is the deciding argument for contracts, HR data or client files.

• Data control: documents, vectors and questions stay on your server.
• Location: hosting in France or Europe.
• Stable cost: one server per month, however many questions are asked.
• Model choice: open models, including Mistral, a French publisher.

Our guide to hosting an AI application details the infrastructure options.

What architecture for a self-hosted RAG?

A simple architecture fits on a single GPU server: Ollama for the language model and embeddings, a vector database for the passages, and Open WebUI as the interface with document management. The building blocks are open source and can be deployed in a few hours.

• Open WebUI already includes document search, as its documentation states: it is the fastest way to test.
• For a custom application, call the Ollama API (/api/embed for embeddings) from your code.

What server do you need for RAG?

Sizing depends on the language model, not on the documents: you need a GPU whose VRAM holds the chosen model, while the vector database and interface need few resources. A corpus of a few thousand pages remains modest for a vector database.

• A 7B to 8B model: it weighs under 5 GB in the Ollama library, so an 8 GB card is enough for testing.
• A 27B model: it weighs 17 GB according to the Gemma 3 page, so a 32 GB GPU becomes necessary.
• Interface and database only: a VPS can host them if the model runs elsewhere.

Wayhost GPU rental offers a dedicated NVIDIA V100 32 GB at €0.60 excl. VAT per hour or €199.99 excl. VAT per month, hosted in Cergy, with root access by SSH key (checked on November 6, 2026). For a whole machine, also see the physical server offer.

Which mistakes should you avoid with RAG?

Failures rarely come from the model and almost always from the documents or the chunking: outdated files, passages that are too long or too short, and no access control.

1. Indexing everything without sorting: duplicates and outdated versions produce contradictory answers.
2. Poor chunking: passages that are too short lose context, and passages that are too long drown the information.
3. Ignoring permissions: an employee must not get, through the AI, a document they are not allowed to read.
4. Not showing sources: users must be able to check the passage cited.
5. Not measuring: build a list of typical questions and check the answers regularly.
6. Exposing the server: the Ollama API is never opened to the Internet; only the protected interface is.

How do you get started with RAG quickly?

The fastest way is to deploy Ollama and Open WebUI on a GPU server, load a selection of documents and test with a list of real questions. Allow one day for a first pilot.

1. Pick a scope: a single team, a few dozen up-to-date documents.
2. Rent a GPU and install Ollama, a language model and an embedding model.
3. Install Open WebUI and load the documents into a collection.
4. Test with the questions the team actually asks.
5. Tune the chunking, the number of passages retrieved and the model.
6. Extend to other teams once the answers are reliable.

To choose the long-term server type, read our server hosting guide.

Frequently asked questions

What does RAG mean in AI?

RAG stands for retrieval-augmented generation. The AI first searches for passages in a document base, then writes its answer from them. It therefore answers from your content and not only from its own memory.

Does RAG stop the AI from making things up?

It greatly reduces made-up answers, because the model relies on passages it was given. It does not remove them entirely, especially when documents are incomplete. Showing sources lets you check every answer.

Do you need a GPU for RAG?

The language model that writes the answers runs much better on a GPU. Embeddings and the vector database are lighter. Without a GPU, generation can go through a remote API.

Which documents can you use in RAG?

Most text formats: PDFs, office documents, web pages, knowledge base articles. Scanned documents must first go through character recognition. The cleaner and more current the documents, the better the answers.

Is RAG GDPR compliant?

RAG is a technique; compliance depends on the hosting and the data processed. Hosting it in France on a server you control simplifies the process. Limit documents to what is needed and manage access rights.

What is the difference between RAG and ChatGPT?

ChatGPT answers from what its model has learned, possibly supplemented by files you send it. An internal RAG answers from your document base, indexed on your side. The documents never leave your server.

How long does it take to set up RAG?

A pilot with Ollama and Open WebUI can be set up in a day on a GPU server. The longest part is choosing and cleaning the documents. A production version then needs access rights and quality monitoring.

Key takeaways

• RAG makes the AI answer from your documents, with sources you can check.
• Four building blocks: chunking, embeddings, vector database, language model.
• RAG does not retrain the model: you update a document, not the AI.
• Hosting it yourself keeps sensitive documents on a server in France.
• Ollama, a vector database and Open WebUI on a GPU are enough for a pilot.

To run your RAG on an NVIDIA V100 32 GB hosted in France, discover Wayhost GPU rental.

Article written by Mouhamed BANKOLE, CEO at ITNET Technologies.