If your app already talks to the OpenAI API, you can often point it at your own model with one changed line. vLLM is the most common way to do that. Here's the short version, based on vLLM's own docs.
What "OpenAI-compatible" means
vLLM provides an HTTP server that implements OpenAI's API. Same paths, same request and response shape. Your code sends the same JSON; it just goes to your server instead.
Supported endpoints include:
/v1/chat/completionsand/v1/completions/v1/responses/v1/embeddings/v1/audio/transcriptionsand/v1/audio/translations
Not every model supports every endpoint. Chat needs a model with a chat template, for example.
Step 1: install
The quickstart targets Linux and Python 3.11 to 3.14, and recommends uv:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
The docs also list AMD, Intel, Google TPU and Ascend options, and vLLM-Metal for Apple Silicon Macs.
Step 2: start the server
vllm serve Qwen/Qwen2.5-1.5B-Instruct
By default it listens on http://localhost:8000. Use --host and --port to change that. The server hosts one model at a time. Models download from Hugging Face by default.
Check it's up:
curl http://localhost:8000/v1/models
Step 3: point your app at it
With the official OpenAI Python client, it's just the base URL and key, as in vLLM's docs:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="token-abc123",
)
completion = client.chat.completions.create(
model="NousResearch/Meta-Llama-3-8B-Instruct",
messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message)
For that example, the docs start the server with vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123.
Already using a gateway? The same idea works behind a proxy layer. (If you use LiteLLM, read our LiteLLM supply-chain post first.)
The security catch most tutorials skip
vLLM's docs put this in a warning box: "Do not rely on --api-key alone to secure vLLM."
The --api-key flag (or VLLM_API_KEY) only checks requests under /v1, /v2 and /inference. Other endpoints on the same server, notably /invocations, aren't authenticated and expose the same inference. The docs recommend hardening like a reverse proxy.
Practical rules:
- Bind to localhost or a private network.
- Put a reverse proxy with real auth in front if anything outside needs access.
- Treat the server like a database: not on the public internet.
Our vibe-coding security checklist covers the same habits for the rest of your stack.
Will my GPU fit the model?
That's the real limit. See how much VRAM you need, or rent a GPU if yours is too small. Big models on small machines are a different game; see running a model bigger than your RAM.
