How to Run Local AI Models with Ollama on an 8GB GPU (RTX 3060 Ti Guide)

Why Run AI Models Locally?

Cloud AI APIs are convenient, but they come with real costs — both financial and practical. Every API call sends your data to a third-party server. Rate limits throttle your workflows. And monthly subscription fees add up fast. For developers, researchers, and privacy-conscious users, running AI models locally is the answer.

With an NVIDIA RTX 3060 Ti (8GB VRAM), you can run surprisingly capable large language models entirely on your own hardware. No cloud bills. No data leaving your machine. No rate limits. This guide walks you through the entire process — from installing Ollama to picking the right models for your GPU.

What Is Ollama?

Ollama is an open-source tool that makes it trivially easy to download, manage, and run large language models locally. Think of it as “Docker for AI models” — one command pulls a model, another command starts chatting with it. It handles GPU acceleration automatically, supports dozens of models out of the box, and provides a clean API compatible with the OpenAI SDK.

Key advantages:

• One-command installation — works on Linux, macOS, and Windows
• Automatic GPU detection — uses CUDA on NVIDIA, Metal on Apple Silicon
• Model library — browse and pull from 100+ models at ollama.com/library
• OpenAI-compatible API — drop-in replacement for existing code
• Model quantization — runs larger models by reducing precision to fit your VRAM

Hardware Reality Check: What Can 8GB VRAM Handle?

The RTX 3060 Ti is a sweet spot for local AI. Its 8GB of GDDR6X VRAM can comfortably run 7B-13B parameter models at 4-bit quantization. Here’s what fits:

Key insight: You need roughly 1GB of VRAM per billion parameters at 4-bit quantization (Q4_K_M). An 8GB GPU gives you about 6.5GB usable for the model after OS and context overhead, which means 7B-8B models are your sweet spot.

Step 1: Install Ollama on Linux

On Ubuntu/Debian, installation is a single command:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Check that Ollama service is running
systemctl status ollama

The installer sets up Ollama as a systemd service that starts automatically. On first run, it creates the model storage directory at /usr/share/ollama/.ollama/models/.

For NVIDIA GPU support: Ollama bundles its own CUDA libraries, but you need the NVIDIA driver installed:

# Check your driver
nvidia-smi

# If not installed (Ubuntu):
sudo apt install nvidia-driver-535
sudo reboot

After reboot, verify GPU is detected:

ollama ps
# Should show GPU: NVIDIA GeForce RTX 3060 Ti

Step 2: Pull Your First Model

Downloading a model is as simple as:

# Pull Mistral 7B (recommended starting point)
ollama pull mistral

# Or try Llama 3.1 8B
ollama pull llama3.1

# Or Phi-3 Mini (great for 8GB GPUs)
ollama pull phi3

The first pull downloads several gigabytes. Subsequent pulls are instant if the model is cached. Check your downloaded models:

ollama list
# NAME            ID              SIZE    MODIFIED
# mistral:latest  61e88e882300    4.1 GB  2 minutes ago

Step 3: Run and Chat with Your Model

Interactive chat mode:

ollama run mistral
>>> What is Infrastructure as Code?
Infrastructure as Code (IaC) is the practice of managing and provisioning...
>>> /bye

Single prompt (useful for scripts):

ollama run mistral "Explain Kubernetes in 3 sentences"

Step 4: Use the OpenAI-Compatible API

Ollama exposes a REST API on http://localhost:11434 that’s compatible with the OpenAI Python SDK. This means you can use it as a drop-in replacement in existing code:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # Required but unused
)

response = client.chat.completions.create(
    model="mistral",
    messages=[
        {"role": "system", "content": "You are a helpful Linux sysadmin."},
        {"role": "user", "content": "How do I check disk usage on Ubuntu?"}
    ],
)

print(response.choices[0].message.content)

Or with plain curl:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral",
    "messages": [{"role": "user", "content": "What is 2+2?"}]
  }'

Step 5: Configure Ollama for Your 3060 Ti

By default, Ollama tries to load the entire model into VRAM. For an 8GB GPU, you may need to tune the configuration for larger models:

# Edit the Ollama service config
sudo systemctl edit ollama

# Add environment variables:
[Service]
Environment="OLLAMA_NUM_GPU=1"
Environment="OLLAMA_GPU_OVERHEAD=256"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_CONTEXT_LENGTH=4096"

# Restart
sudo systemctl restart ollama

Key environment variables:

• OLLAMA_NUM_GPU — Number of GPU layers to offload (0 = CPU only, 999 = all)
• OLLAMA_GPU_OVERHEAD — MB of VRAM to reserve for CUDA overhead
• OLLAMA_MAX_LOADED_MODELS — How many models to keep in memory at once
• OLLAMA_CONTEXT_LENGTH — Max context window (higher = more VRAM)

If a model doesn’t fit entirely in VRAM, Ollama automatically offloads excess layers to system RAM. It’s slower but still functional — a 13B model with partial GPU offload on a 3060 Ti runs at about 60-70% of full-GPU speed.

Best Models for Agent/Automation Tasks

If you’re using Ollama with an AI agent framework (like Hermes Agent), you want models that follow instructions well and support tool calling:

Top Picks for 8GB GPUs

1. Hermes 3 (8B) — Fine-tuned for agent tasks, tool use, and structured output. Excellent instruction following. ollama pull hermes3

2. Mistral 7B Instruct — Strong all-around performer. Great reasoning, good code generation. ollama pull mistral

3. Llama 3.1 8B Instruct — Meta’s latest. Excellent multilingual support and reasoning. ollama pull llama3.1

4. Phi-3 Mini (3.8B) — Microsoft’s compact model. Surprisingly capable for its size. Perfect if you need headroom for larger context. ollama pull phi3

5. CodeLlama 7B — If your primary use case is code generation and review. ollama pull codellama

Models to Avoid on 8GB

• Models larger than 13B at Q4 quantization (won’t fit)
• Unquantized (F16) models above 7B (double the VRAM requirement)
• Yi-34B, Hermes-2-Yi-34B (need 20GB+ VRAM)

Integrating Ollama with Hermes Agent

If you’re using Hermes Agent, you can configure it to use your local Ollama instance instead of cloud APIs:

# Configure Hermes to use local Ollama
hermes config set model.default hermes3:latest
hermes config set model.provider ollama
hermes config set model.base_url http://localhost:11434/v1

This gives you a fully local AI agent — no API keys, no cloud costs, no data leaving your machine. The agent can browse the web, manage files, run commands, and automate tasks, all powered by your local GPU.

Performance Tips for RTX 3060 Ti

Get the most out of your 8GB:

1. Use Q4_K_M quantization — Best quality-to-size ratio. This is the default for most Ollama models.
2. Reduce context length — OLLAMA_CONTEXT_LENGTH=2048 saves ~1GB VRAM vs 4096. Increase only when needed.
3. Run one model at a time — Set OLLAMA_MAX_LOADED_MODELS=1 to avoid VRAM contention.
4. Close other GPU apps — Browsers with hardware acceleration, games, and video editors compete for VRAM.
5. Keep drivers updated — NVIDIA driver 535+ has better memory management for compute workloads.
6. Use ollama ps — Monitor which models are loaded and how much VRAM they’re using.

Troubleshooting Common Issues

“out of memory” errors: The model is too large for your GPU. Try a smaller model (3B-7B), reduce context length, or increase OLLAMA_GPU_OVERHEAD.

Slow generation (< 10 tokens/sec): Check nvidia-smi — if GPU utilization is low, the model may be partially offloaded to CPU. Reduce OLLAMA_NUM_GPU to force full GPU loading, or use a smaller model.

Ollama doesn’t detect GPU: Reinstall NVIDIA drivers and reboot. Verify with nvidia-smi. On some systems, you may need to set OLLAMA_NUM_GPU=999 explicitly.

Model gives poor responses: Try a different quantization level. Q4_K_M is the sweet spot. Q8_0 gives better quality but uses more VRAM. Q2_K is fastest but noticeably worse.

Conclusion

Running AI models locally on an RTX 3060 Ti is not just possible — it’s practical. With Ollama, you can have a capable AI assistant running in minutes, with zero recurring costs and complete privacy. The 7B-8B parameter range hits the sweet spot for 8GB VRAM, offering strong reasoning, code generation, and instruction following.

Whether you’re building AI agents, automating workflows, or just experimenting, local AI puts the power of large language models directly on your desk. No cloud required.

Got questions about running Ollama on your setup? Drop a comment below.