ollama37/docker/README.md

Ollama37 Docker Build System

Docker infrastructure for running Ollama on NVIDIA Tesla K80 GPUs (CUDA Compute Capability 3.7).

Overview

This directory contains Docker build configurations for the ollama37 project, which maintains support for legacy NVIDIA Tesla K80 GPUs. The official Ollama project dropped support for Compute Capability 3.7 when transitioning to CUDA 12, but this fork preserves compatibility using CUDA 11.4.

Directory Structure

docker/
├── README.md                  # This file - overview of Docker build system
├── docker-compose.yml         # Simplified deployment configuration
├── builder/                   # Build environment image
│   ├── Dockerfile             # Builder base image (CUDA 11.4 + GCC 10 + Go 1.24)
│   ├── README.md              # Builder documentation
│   └── scripts/               # Environment setup scripts
│       ├── cuda-11.4.sh       # CUDA 11.4 PATH configuration
│       ├── gcc-10.sh          # GCC 10 library paths
│       └── go-1.24.2.sh       # Go 1.24.2 PATH configuration
└── runtime/                   # Production runtime image
    ├── Dockerfile             # Multi-stage build for ollama37 binary
    └── README.md              # Runtime image documentation

Components

Builder Image (builder/)

The builder image provides a complete compilation environment for building ollama37 from source with Tesla K80 support.

Base: Rocky Linux 8
Key Software:

• CUDA 11.4 Toolkit (last version supporting Compute Capability 3.7)
• NVIDIA Driver 470 (compatible with Tesla K80)
• GCC 10 (custom-built from source)
• CMake 4.0.0
• Go 1.24.2

Purpose: Compile ollama37 binary with optimized CUDA kernels for Tesla K80 GPUs.

See builder/README.md for detailed information.

Runtime Image (runtime/)

The runtime image is a minimal production image containing only the compiled ollama37 binary and required CUDA libraries.

Base: Rocky Linux 8
Build: Multi-stage build using builder image
Size: Optimized for production deployment

Purpose: Run ollama37 server with NVIDIA GPU acceleration.

See runtime/README.md for usage instructions.

Quick Start

Option 1: Build from Source (Recommended for Development)

Build both builder and runtime images locally:

cd /home/jack/src/ollama37/docker

# Build the builder image first (takes ~30-60 minutes)
docker build -t ollama37-builder:local -f builder/Dockerfile builder/

# Build the runtime image (uses local builder)
docker build -t ollama37:local -f runtime/Dockerfile runtime/

# Run with docker-compose
docker-compose up -d

Option 2: Use Pre-built Images

Pull and run pre-built images from Docker Hub:

cd /home/jack/src/ollama37/docker
docker-compose pull
docker-compose up -d

Option 3: Manual Docker Run

Run the runtime image directly:

docker run -d \
  --name ollama37 \
  --runtime nvidia \
  -p 11434:11434 \
  -v ollama37-data:/root/.ollama \
  ollama37:local

Usage

Start the Server

docker-compose up -d

Check Logs

docker-compose logs -f

Pull a Model

docker exec -it ollama37 ollama pull llama3.2:3b

Run a Chat Session

docker exec -it ollama37 ollama run llama3.2:3b

API Access

The Ollama API is available at http://localhost:11434:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Why is the sky blue?"
}'

Stop the Server

docker-compose down

Tesla K80 Support

Hardware Requirements

• GPU: NVIDIA Tesla K80 (Compute Capability 3.7)
• VRAM: 12GB per GPU (24GB for dual-GPU K80)
• Driver: NVIDIA Driver 470 or compatible
• Runtime: nvidia-docker2 or NVIDIA Container Toolkit

Recommended Models

Based on 12GB VRAM per GPU:

Model	Size	Quantization	Context Length
Llama 3.2 3B	3B	Full precision	8K
Qwen 2.5 7B	7B	Q4_K_M	4K
Llama 3.1 8B	8B	Q4_0	4K
Mistral 7B	7B	Q4_K_M	4K

For larger models, use aggressive quantization (Q4_0, Q4_K_M) or multi-GPU setups.

Multi-GPU Support

Tesla K80 dual-GPU configurations are supported:

# Use both GPUs
docker run --gpus all ...

# Use specific GPU
docker run --gpus '"device=0"' ...

# Split model across GPUs
docker run -e CUDA_VISIBLE_DEVICES=0,1 ...

Build Configuration

Environment Variables

The runtime Dockerfile uses these build arguments:

• CC=/usr/local/bin/gcc - Use custom GCC 10
• CXX=/usr/local/bin/g++ - Use custom G++ 10

CUDA Architecture Targets

The build includes Compute Capability 3.7 in CMAKE_CUDA_ARCHITECTURES:

set(CMAKE_CUDA_ARCHITECTURES "37;50;61;70;75;80")

This ensures CUDA kernels are compiled for Tesla K80 (CC 3.7).

Troubleshooting

GPU Not Detected

Check NVIDIA runtime and driver installation:

docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi

Out of Memory Errors

Reduce context length or use more aggressive quantization:

docker exec ollama37 ollama run llama3.2:3b --num-ctx 2048

Build Failures

Ensure sufficient disk space and memory:

• Disk space: 50GB+ recommended
• RAM: 16GB+ recommended for building
• Swap: 8GB+ recommended if RAM is limited

Development

Rebuilding After Changes

# Rebuild builder image
docker build --no-cache -t ollama37-builder:local -f builder/Dockerfile builder/

# Rebuild runtime image
docker build --no-cache -t ollama37:local -f runtime/Dockerfile runtime/

Testing Local Builds

# Run with local image
docker run --rm -it --gpus all ollama37:local ollama --version

Contributing

See the main project repository for contribution guidelines:

• ollama37 GitHub Repository

License

This project maintains the same license as the upstream Ollama project.

Resources

• Ollama Documentation
• CUDA 11.4 Documentation
• Tesla K80 Specifications
• NVIDIA Container Toolkit