
vLLM serves a language model to many users behind an OpenAI-compatible API. How it works, differences with Ollama, compatible GPUs (compute capability 7.5 minimum), memory needs, installation, security and alternatives.
By Mouhamed BANKOLE, CEO at ITNET Technologies · Updated November 18, 2026 · Reading time: 10 min
You have a language model running nicely on your workstation with Ollama, and now you want to offer it to a whole team, or put it behind an application, without responses slowing down as soon as several people use it. vLLM is the tool most often mentioned for this move to production. You still need to know what it really does, which GPUs it runs on today and when you are better off without it. This guide answers those questions, with a GPU compatibility grid, an installation procedure and the alternatives if your card is not supported.
In short
- vLLM is an open source inference engine that serves a language model to many simultaneous users, behind an OpenAI-compatible API.
- Its gains come from memory management (PagedAttention) and continuous batching of requests: it raises total throughput, not the speed of a single conversation.
- The stable release requires an NVIDIA GPU with compute capability 7.5 or higher on Linux: a T4, an RTX 3060 or an A100 qualify, a V100 does not.
- Ollama remains simpler for a workstation or small internal use; vLLM becomes useful when concurrent requests pile up.
- Without a compatible GPU, you have two options: Ollama on your current card, or an OpenAI-compatible AI API that spares you server administration.
Contents
- What is vLLM?
- How does vLLM speed up a language model?
- vLLM or Ollama: which should you choose?
- Which GPU do you need to run vLLM?
- How much GPU memory does a model need?
- How do you install vLLM and start an OpenAI-compatible API?
- How do you secure a vLLM server in production?
- What if your GPU is not compatible with vLLM?
- Should you rent a GPU server or use an AI API?
What is vLLM?
vLLM is an open source library that serves large language models (LLMs) efficiently, with an inference server that receives requests and an engine that optimizes computation on the GPU (Red Hat). In practice, you give it a model, it loads it into memory and answers requests from your applications.
What you need to know up front:
- it is server software, designed for Linux and datacenter GPUs, not a desktop application;
- it downloads models from Hugging Face by default (vLLM documentation);
- it exposes an OpenAI-compatible API: a tool already connected to the OpenAI API only needs a new address;
- a vLLM server serves one model at a time (vLLM documentation);
- it is maintained by a large community, with contributors such as UC Berkeley, Hugging Face, NVIDIA and Red Hat (Red Hat).
In other words, vLLM does not replace the model: it replaces the layer that runs it and makes it available on the network.
How does vLLM speed up a language model?
vLLM mainly speeds up serving many requests in parallel, thanks to two techniques: PagedAttention for memory and continuous batching for computation. A single conversation is not necessarily faster; it is the server's total throughput that goes up.
The mechanisms involved:
- PagedAttention splits the model's working memory (the key-value cache, or KV cache) into pages, on the same principle as an operating system's virtual memory; the technique was described in a 2023 research paper;
- less wasted memory means more conversations held at the same time on the same GPU;
- continuous batching adds a new request to the running batch as soon as a slot frees up, instead of waiting for the batch to finish;
- prefix caching reuses the computation of a shared prompt opening, which helps when all requests share the same system instructions.
On the resulting effect, Red Hat cites throughput up to 24 times higher than Hugging Face Transformers, based on a 2023 post by the vLLM team (Red Hat). That figure is the project's own measurement on its own tests: measure on your model and your traffic before sizing anything.
Pro tip: judge vLLM on the number of requests served per second and on time to first token under load, not on the speed of a single answer.
vLLM or Ollama: which should you choose?
Choose Ollama for a workstation or small internal use, and vLLM when several users or an application query the model at the same time. The two tools do not solve the same problem.
| Criterion | Ollama | vLLM |
|---|---|---|
| Main use | Workstation, prototype, small team | Shared service, application, API |
| Installation | One installer, a model in one command | Python environment or container, tuning required |
| Model formats | Quantized GGUF, Ollama library | Hugging Face weights, GPU quantized formats |
| Concurrent requests | Fine at low load | Built for load |
| Supported NVIDIA GPUs | Compute capability 5.0 and up (Ollama documentation) | Compute capability 7.5 and up (vLLM documentation) |
| Operating systems | Linux, Windows, macOS | Linux (no native Windows) |
| API | Its own API and an OpenAI-compatible API | OpenAI-compatible API |
Signs that it is time to move to vLLM:
- latency becomes erratic as soon as several people ask questions at once;
- the GPU is underused while requests wait their turn;
- the model no longer fits on a single GPU and must be split across several cards;
- you need metrics to track load and quality of service.
For installing Ollama and choosing a model based on available memory, our Ollama guide walks through each step. Many teams keep Ollama for development and reserve vLLM for the shared server.
Which GPU do you need to run vLLM?
The stable release of vLLM requires an NVIDIA GPU with compute capability 7.5 or higher, for example a T4, an RTX 20 series card, an A100, an L4, an H100 or a B200 (vLLM documentation). The same page states that vLLM does not support Windows natively.
Compute capability is a generation number that NVIDIA assigns to each GPU. Several online guides still give a minimum of 7.0: that threshold is out of date, and it is what makes installations fail on Volta-generation cards.
| GPU | Compute capability | Stable vLLM | Source |
|---|---|---|---|
| NVIDIA V100 | 7.0 | Not supported | NVIDIA |
| NVIDIA T4 | 7.5 | Supported | NVIDIA |
| GeForce RTX 3060 | 8.6 | Supported | NVIDIA |
| NVIDIA A100 | 8.0 | Supported | NVIDIA |
| NVIDIA L4 | 8.9 | Supported | NVIDIA |
| NVIDIA H100 and H200 | 9.0 | Supported | NVIDIA |
| NVIDIA B300 | 10.3 | Supported | NVIDIA |
Before renting or buying a card, check three things:
- the compute capability in NVIDIA's official list;
- the GPU memory, which determines the model size you can run (see the next section);
- the operating system: a recent Linux distribution with an up-to-date NVIDIA driver.
Our article on the NVIDIA V100 explains what this card still does well and why it falls outside vLLM's scope.
How much GPU memory does a model need?
Plan at least for the size of the model weights, plus headroom for the conversation cache: in BF16, each parameter takes 2 bytes, so an 8-billion-parameter model needs about 16 GB for its weights alone (BF16 format, stored on 16 bits). The calculation is simple: number of parameters × 2 bytes in BF16, × 1 byte in FP8.
What uses GPU memory:
- the model weights, loaded once at startup;
- the KV cache, which grows with context length and the number of concurrent conversations;
- the working reserve of vLLM and the CUDA libraries;
- a safety margin, to avoid out-of-memory errors under load.
| Model size | Weights in BF16 (2 bytes/parameter) | Weights in FP8 (1 byte/parameter) | Example of a suitable GPU |
|---|---|---|---|
| 1.5 billion parameters | ≈ 3 GB | ≈ 1.5 GB | RTX 3060 12 GB |
| 8 billion | ≈ 16 GB | ≈ 8 GB | A100 80 GB in BF16, RTX 3060 12 GB with 4-bit quantization (AWQ, GPTQ) |
| 32 billion | ≈ 64 GB | ≈ 32 GB | A100 or H100 80 GB |
| 70 billion | ≈ 140 GB | ≈ 70 GB | H200 141 GB in FP8, or several GPUs |
Two settings keep consumption under control:
--max-model-lensets the maximum context length: start from a realistic value rather than the model's maximum;--gpu-memory-utilizationsets the share of GPU memory that vLLM is allowed to reserve.
Pro tip: a quantized model (AWQ, GPTQ or FP8) sharply reduces memory needs. Test answer quality on your own cases before adopting it in production.
How do you install vLLM and start an OpenAI-compatible API?
Install vLLM in a dedicated Python environment with uv pip install vllm --torch-backend=auto, then start the server with vllm serve followed by the model name: the API answers on http://localhost:8000 by default (vLLM documentation).
The procedure on a freshly installed Linux GPU server:
- check the GPU with
nvidia-smi: the driver must respond and show available memory; - create an isolated Python environment with
uv venv, then activate it; - install vLLM with
uv pip install vllm --torch-backend=auto; - start the server with a small model for testing, for example
vllm serve Qwen/Qwen2.5-1.5B-Instruct; - require an API key with the
--api-keyoption or theVLLM_API_KEYvariable (vLLM documentation); - test the API:
curl http://localhost:8000/v1/modelsshould list the loaded model; - connect your tools by replacing the OpenAI API address with your server's.
The available endpoints:
/v1/modelsto list the served model;/v1/chat/completionsfor conversations, the most widely used format;/v1/completionsfor plain text completion.
By default, the server applies the chat template shipped with the model's tokenizer (vLLM documentation). If answers look inconsistent after migrating from Ollama, that template is often the first thing to check.
Pro tip: an OpenAI-compatible API is not an exact copy. Test the advanced features you depend on (tool calls, structured outputs, streaming) before switching an application over.
How do you secure a vLLM server in production?
Never leave the vLLM port open to the Internet: put it behind an HTTPS proxy, require an API key and restrict allowed addresses. An exposed inference server is free GPU compute for anyone, and sometimes leaked data.
Checklist before going live:
- listen on the local interface and let a proxy (Caddy, nginx or Traefik) serve the API over HTTPS;
- enable an API key and rotate it regularly: vLLM accepts several keys at once, which makes rotation easier (vLLM documentation);
- filter with a firewall: only SSH, HTTP and HTTPS stay open;
- pin the vLLM and model versions, and test every update on a staging machine;
- monitor memory and latency to spot saturation before your users do;
- log without keeping prompt contents if they contain personal data.
For a chat interface connected to your server, our Open WebUI guide shows how to set up a private ChatGPT in front of an OpenAI-compatible API.
What if your GPU is not compatible with vLLM?
If your GPU is below compute capability 7.5, use Ollama or another llama.cpp-based engine on the same card, or switch to a hosted OpenAI-compatible API. Forcing an old vLLM release onto it cuts you off from fixes and recent models.
The options, from simplest to most demanding:
- Ollama on your current GPU: it supports NVIDIA cards from compute capability 5.0 (Ollama documentation), V100 included;
- an OpenAI-compatible AI API: your applications keep the same code, only the address changes;
- a more recent GPU: an RTX 3060 is enough to test vLLM with a small model, an A100 or H100 for production;
- the CPU: possible for experiments, too slow for a shared service.
| Situation | Recommended option | Why |
|---|---|---|
| Internal assistant, a few users, V100 | Ollama | Supported, easy to run |
| Application that already calls the OpenAI API | OpenAI-compatible AI API | No server to manage, same code |
| High-traffic shared service | vLLM on A100, H100 or H200 | Throughput under load |
| Testing vLLM on a small budget | vLLM on RTX 3060 with a small model | Compute capability 8.6 |
Should you rent a GPU server or use an AI API?
Rent a GPU server if you want to choose your model, control the configuration and keep steady throughput; use an API if you only want to add AI to an application without running a server. The right choice depends on volume and on your operations skills.
The Wayhost GPU offer includes V100 32 GB, RTX 3060 12 GB, A100 80 GB, H100 80 GB, H200 141 GB and B300 288 GB cards, hosted in Cergy, France, with root access via SSH key (checked on November 18, 2026). On pricing, the dedicated RTX 3060 is listed at €0.20 excl. VAT per hour or €99 excl. VAT per month, the A100 80 GB at €799 excl. VAT per month and the H100 80 GB at €1,521 excl. VAT per month (checked on November 18, 2026). According to the compatibility grid above, all of these cards suit vLLM except the V100, which is better matched with Ollama.
The same page presents an OpenAI-compatible AI API, charged per token against prepaid credit, whose requests are not used to train models (checked on November 18, 2026). It is the direct alternative when you do not need to run the server yourself.
Criteria to decide:
- volume: a per-token API suits irregular usage, a rented GPU becomes worthwhile when it runs continuously;
- model choice: a server lets you load any compatible model, an API imposes its catalog;
- skills: a vLLM server requires monitoring, updates and network security;
- data: in both cases, check where it is processed and whether it is retained.
For the rest of the application (database, web interface, task queues), a VPS separate from the GPU is often enough, and a physical server takes over for heavy workloads. Our guide to hosting an AI application helps you size the whole stack, and our article on the dedicated server compares infrastructure options.
Frequently asked questions
What is vLLM in a few words?
vLLM is open source software that runs a language model on a GPU server and makes it available through an OpenAI-compatible API. It is designed to serve many requests at the same time. It is used to put an LLM into production.
Is vLLM free?
Yes, the software is open source and free. You do pay for the hardware that runs it, in practice a server with a compatible NVIDIA GPU. Each model you use has its own license, which you should check before commercial use.
Is vLLM faster than Ollama?
With many simultaneous users, vLLM generally serves more requests per second thanks to its memory management and continuous batching. For a single person, the difference is small and Ollama is simpler. Measure on your own traffic before migrating.
Does vLLM run on Windows?
Not natively: the official documentation states that vLLM does not support Windows. You need a Linux server, or WSL on a Windows workstation for experiments. In production, a Linux GPU server is the reference setup.
Can you use vLLM on an NVIDIA V100?
Not with the current stable release, which requires compute capability 7.5 or higher, while the V100 is 7.0. On a V100, use Ollama instead, which supports it. For vLLM, choose a T4, an RTX 3060, an A100 or newer.
How much GPU memory does vLLM need?
It depends on the model: count 2 bytes per parameter in BF16 for the weights, plus headroom for the conversation cache. An 8-billion-parameter model therefore needs about 16 GB for its weights alone, since the BF16 format stores each parameter on 16 bits. Quantization reduces that need significantly.
Is vLLM compatible with the OpenAI API?
Yes, vLLM exposes OpenAI API endpoints, including chat and completions. Most OpenAI-compatible tools work once you change the API address and key. Still, check the advanced features your application relies on.
Can one vLLM server serve several models?
A vLLM process serves one model at a time. To offer several models, you run several instances, on several GPUs or on different ports, behind a proxy that routes requests. Each instance needs its own GPU memory.
Key takeaways
- vLLM serves an LLM at scale behind an OpenAI-compatible API, with throughput designed for load.
- It requires an NVIDIA GPU with compute capability 7.5 or higher on Linux: the V100 is no longer one of them.
- Ollama remains the right choice for a workstation or small internal use, and on older GPUs.
- Memory determines model size: 2 bytes per parameter in BF16, plus the conversation cache.
- Secure the API with an HTTPS proxy, a key and a firewall before going live.
To run vLLM on an A100, H100 or H200 hosted in France, or to use an OpenAI-compatible AI API directly, discover the Wayhost GPU offer.
Article written by Mouhamed BANKOLE, CEO at ITNET Technologies.