Code
cookbook/11_models/vllm/basic.py
Usage
1
Set up your virtual environment
2
Setup vLLM Server
Start a vLLM server locally:
3
Install dependencies
4
Run Agent
Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
from agno.agent import Agent
from agno.models.vllm import VLLM
agent = Agent(
model=VLLM(id="Qwen/Qwen2.5-7B-Instruct", top_k=20, enable_thinking=False),
markdown=True,
)
agent.print_response("Share a 2 sentence horror story")
Set up your virtual environment
uv venv --python 3.12
source .venv/bin/activate
uv venv --python 3.12
.venv\Scripts\activate
Setup vLLM Server
uv pip install vllm
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct \
--port 8000
Install dependencies
uv pip install -U openai agno
Run Agent
python cookbook/11_models/vllm/basic.py
Was this page helpful?