Ollama runs open AI models (Llama, Mistral, Gemma) locally. Installation, commands, choosing a model for your VRAM, checking GPU use, moving to a rented GPU server and exposing the API safely.
By Mouhamed BANKOLE, CEO at ITNET Technologies · Updated November 6, 2026 · Reading time: 9 min
You want an AI assistant without sending your documents to an online service, or you want to test open models without paying per user. Ollama is the simplest tool for that: it downloads a model and runs it on your machine with one command. You still need to know which model fits your graphics card, how to check that it really uses the GPU, and what to do when your PC is no longer enough. This guide answers those questions, up to deploying Ollama on a rented GPU server and securing its API.
In short
- Ollama is a free, open-source tool (MIT license) that downloads and runs open AI models (Llama, Mistral, Gemma…) on macOS, Windows or Linux.
- It installs with one command and is driven with
ollama run,ollama pull,ollama psor through its API on port 11434.- Video memory (VRAM) is the limiting factor: the model must fit on the GPU, otherwise part of it falls back to the CPU and slows down sharply.
- Beyond a PC, a rented GPU server (for example an NVIDIA V100 32 GB) runs larger models, around the clock, for a whole team.
- Never expose port 11434 to the Internet: the API has no built-in authentication, so use an SSH tunnel or a reverse proxy with access control.
Contents
- What is Ollama?
- How do you install Ollama?
- Which Ollama commands should you know?
- Which model should you pick for your graphics card's memory?
- How do you check that Ollama really uses the GPU?
- When is a PC no longer enough to run Ollama?
- How do you install Ollama on a rented GPU server?
- How do you expose the Ollama API safely?
- How much does local AI cost on a rented GPU?
What is Ollama?
Ollama is free software that downloads open language models and runs them on your own machine, with a command-line interface and a local API. The project is published under the MIT license on GitHub and runs on macOS, Windows, Linux and in Docker.
What it gives you compared with an online AI service:
- privacy: requests and documents stay on your machine or server, ideally hosted in France;
- cost: no per-user subscription, only the hardware or the GPU rental;
- model choice: Llama, Mistral (a French publisher), Gemma, Qwen and many more in the Ollama library;
- integration: a local API that tools such as Open WebUI, n8n or your own scripts can call.
If you are choosing the infrastructure for a complete AI project instead, our guide to hosting an AI application compares the options.
How do you install Ollama?
On Linux one command is enough; on macOS and Windows you install the app from the official website. The Linux command is the one given in the official Ollama repository:
curl -fsSL https://ollama.com/install.sh | sh
Steps for your system:
- Linux: run the command above; Ollama installs as a system service.
- macOS and Windows: download the installer from ollama.com, then open the app.
- Docker: use the official
ollama/ollamaimage, handy on a server. - Check:
ollama --versionshould print the installed version.
Pro tip: on Linux, install the NVIDIA driver before Ollama. The installer then detects the GPU and uses it with no extra setup.
Which Ollama commands should you know?
Five commands cover the essentials: run to chat with a model, pull to download it, list to see installed models, ps to see loaded models and stop to unload one.
| Command | What it does |
|---|---|
ollama run mistral | Downloads the model if needed and opens a chat |
ollama pull llama3.1:8b | Downloads a model without starting it |
ollama list | Lists installed models |
ollama ps | Shows loaded models and whether they run on GPU or CPU |
ollama stop mistral | Unloads the model from memory |
Two useful settings, documented in the official Ollama FAQ:
- keep-alive time: a model stays loaded for 5 minutes after the last request, configurable with
OLLAMA_KEEP_ALIVE; - context size: 4096 tokens by default, configurable with
OLLAMA_CONTEXT_LENGTH, at the cost of more memory.
Which model should you pick for your graphics card's memory?
Pick a model whose download size fits in your card's VRAM with room left for context: about 4 to 5 GB for a 7B to 8B model, 17 GB for a 27B model, and over 40 GB for a 70B model. The sizes below are those shown in the Ollama library on November 6, 2026 (Gemma 3, Mistral, Llama 3.1).
| Model | Listed size | 8 GB card | 16 GB card | 32 GB GPU |
|---|---|---|---|---|
| gemma3:4b | 3.4 GB | Yes | Yes | Yes |
| mistral (7b) | 4.4 GB | Yes | Yes | Yes |
| llama3.1:8b | 4.9 GB | Yes, short context | Yes | Yes |
| gemma3:12b | 8.2 GB | No | Yes | Yes |
| gemma3:27b | 17 GB | No | No | Yes |
| llama3.1:70b | 43 GB | No | No | No (2 GPUs) |
- The listed size is a quantized model (compressed weights), which makes it usable on consumer cards.
- Context also uses memory: the more you raise
OLLAMA_CONTEXT_LENGTH, the more headroom you need. - A bigger model is not always better for your use case: start with a 7B or 8B model such as Mistral.
How do you check that Ollama really uses the GPU?
Run ollama ps while a model is answering: the PROCESSOR column should read "100% GPU". The Ollama FAQ explains that a value such as "48%/52% CPU/GPU" means the model overflows the VRAM and part of it runs on the CPU.
If the GPU is not used:
- check the driver with
nvidia-smi; - check compatibility: the Ollama GPU page requires an NVIDIA card with compute capability 5.0 or higher and driver 550 or newer;
- shrink the model or the context if the display shows a CPU/GPU split;
- limit the GPUs used with
CUDA_VISIBLE_DEVICESon a machine that has several.
When is a PC no longer enough to run Ollama?
A PC hits its limits as soon as you need a model larger than its VRAM, a service that is always available, or several users at once. That is when to move to a GPU server.
The telltale signs:
- the target model exceeds the card's VRAM (beyond a 12B model on an 8 GB card, based on Gemma 3 sizes);
- the service must stay reachable even when the workstation is off or you are travelling;
- several colleagues need to query the same model;
- the noise and heat of a fully loaded GPU get in the way at the office.
A GPU server is managed like a dedicated server: SSH access, updates, firewall. For a web front end in front of Ollama without a dedicated GPU, a VPS can host the interface and call a remote API; our guide to comparing VPS plans helps you size it.
How do you install Ollama on a rented GPU server?
Connect over SSH, check the GPU with nvidia-smi, install Ollama with the official command, then pull your model: allow about fifteen minutes. The procedure is the same as on a Linux PC.
- Connect:
ssh root@server-address, with your SSH key. - Check the card:
nvidia-smishould show the GPU and its memory. - Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh. - Pull a model:
ollama pull gemma3:27bon a 32 GB GPU. - Test:
ollama run gemma3:27b, thenollama psto confirm "100% GPU", as the Ollama FAQ explains.
Wayhost GPU rental offers NVIDIA V100 32 GB cards hosted in France, in Cergy, with root access by SSH key, by the hour or by the month (checked on November 6, 2026). The V100 is on the list of compatible cards in the Ollama GPU documentation (compute capability 7.0).
How do you expose the Ollama API safely?
Keep Ollama listening on 127.0.0.1 and reach it through an SSH tunnel or a reverse proxy with authentication: never open port 11434 to the Internet. By default Ollama only listens on the local machine, and its FAQ describes no built-in API authentication.
Three methods, from simplest to most complete:
| Method | For whom | Setup |
|---|---|---|
| SSH tunnel | Just you, from your workstation | ssh -L 11434:localhost:11434 root@server |
| Reverse proxy (nginx) + authentication | A team or an application | HTTPS, password or key, IP filtering |
| Web interface (Open WebUI) | Non-technical users | User accounts, Ollama stays local |
- Only set
OLLAMA_HOST=0.0.0.0behind a firewall that blocks port 11434 from outside. - The official documentation provides a sample nginx configuration with
proxy_pass http://localhost:11434. - Add HTTPS with a Let's Encrypt certificate on your domain name, once you have linked it to your server.
How much does local AI cost on a rented GPU?
The cost is simple to work out: hours of use multiplied by the hourly rate, or a monthly plan if the service runs around the clock. There is no per-token billing when you run Ollama yourself.
On the Wayhost GPU page, on November 6, 2026, we saw:
- 1 dedicated V100 SXM2 32 GB GPU: €0.60 excl. VAT per hour or €199.99 excl. VAT per month, with 8 vCPUs, 48 GB of RAM and 200 GB of storage;
- 2 dedicated V100 32 GB GPUs: €1.20 excl. VAT per hour or €399 excl. VAT per month, for larger models;
- a V100 plan with an 8 GB VRAM budget at €0.18 excl. VAT per hour, shown as "Unavailable" that day;
- prepaid hours in blocks, with no commitment.
For a heavier or permanent workload, also compare with a physical server and read our server hosting guide.
Frequently asked questions
Is Ollama free?
Yes, Ollama is free software released under the MIT license. The open models it downloads are free too, each with its own license. The only cost is the hardware or the server rental.
Does Ollama work without a graphics card?
Yes, Ollama can run on the CPU alone. Answers are then much slower, especially with models above a few billion parameters. A compatible NVIDIA GPU speeds up generation considerably.
Which graphics card do you need for Ollama?
An NVIDIA card with compute capability 5.0 or higher is supported, according to the Ollama documentation. In practice, 8 GB of VRAM is enough for a 7B to 8B model, which weighs under 5 GB in the Ollama library. A 27B model needs a 32 GB card, based on the Gemma 3 page.
Does my data stay private with Ollama?
Yes, as long as the model runs on your machine or server, requests do not go to a vendor. Be careful not to expose the API without protection. On a rented server, choose a host in France or Europe.
What is the difference between Ollama and ChatGPT?
ChatGPT is an online service that runs OpenAI's models on its own servers. Ollama runs open models on your own hardware. You gain privacy, but you manage the computing power yourself.
Can you use Ollama with a web interface?
Yes, interfaces such as Open WebUI plug into the Ollama API. They offer an experience close to ChatGPT, with user accounts. Ollama then stays reachable only locally on the server.
How do you update Ollama?
On Linux, run the official install command again; it replaces the existing version. On macOS and Windows, the app offers the update. Remember to restart the service afterwards.
Where does Ollama store models?
On Linux in /usr/share/ollama/.ollama/models, and in the .ollama folder of the user profile on macOS and Windows. The OLLAMA_MODELS variable changes that location. Plan enough disk space for large models.
Key takeaways
- Ollama runs open AI models locally, with a one-line install and an API.
- VRAM decides the model: 8 GB for a 7B-8B model, 32 GB for a 27B, based on Ollama library sizes.
ollama psshows whether the GPU is working: aim for "100% GPU" (Ollama FAQ).- A rented GPU server takes over when the PC is not enough, for a bigger model or a team.
- Port 11434 is never opened to the Internet: use an SSH tunnel or a reverse proxy with authentication.
To run Ollama on an NVIDIA V100 32 GB hosted in France, discover Wayhost GPU rental.
Article written by Mouhamed BANKOLE, CEO at ITNET Technologies.