ollama37/docs/manual-build.md

Manual Build Guide for Ollama37

This document provides comprehensive instructions for building Ollama37 from source on various platforms, specifically optimized for Tesla K80 and CUDA Compute Capability 3.7 hardware.

⚠️ Important: Kernel Compatibility Notice

Recent kernel updates in Fedora, Ubuntu, and Rocky Linux have broken compatibility with:

• NVIDIA Driver 470 (required for Tesla K80 / Compute Capability 3.7)
• CUDA 11.4 nvcc compiler

Solution: Compile a compatible kernel from source (Linux 5.14.x) before installing NVIDIA drivers.

Recommended Linux Distribution: Rocky Linux 9

• Rocky Linux 8 has docker-ce compatibility issues
• Rocky Linux 9 provides better stability and container support

---

Native Build Overview

For native builds on Rocky Linux 9, you'll need to follow these steps in order:

Installation Steps:

1. Install GCC 10 - Required for kernel compilation and ollama37 source builds
2. Compile Custom Kernel (Linux 5.14.x) - Required for NVIDIA 470 compatibility
3. Install NVIDIA Driver 470 - Tesla K80 GPU driver support
4. Install CUDA 11.4 Toolkit - CUDA development environment
5. Install CMake 4.0 - Build system
6. Install Go 1.25.3 - Go compiler
7. Compile Ollama37 (Optional - if not using pre-built binaries)

Quick Native Build (after prerequisites):

# Clone repository
git clone https://github.com/dogkeeper886/ollama37
cd ollama37

# If compiling from source (requires GCC 10):
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -B build
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake --build build -j$(nproc)
go build -o ollama .

# If using pre-built binary (GCC 10 not required):
# Just download and run the ollama binary

---

Detailed Installation Guide for Rocky Linux 9

Step 1: GCC 10 Installation

Why Install GCC 10 First?

GCC 10 is required for:

• Compiling the custom Linux kernel (Step 2)
• Building ollama37 from source (Step 7)
• CUDA 11.4 compatibility (CUDA 11.4 nvcc is not compatible with GCC 11.5+)

Rocky Linux 9 ships with GCC 11.5 by default, which is:

• ❌ Incompatible with CUDA 11.4 nvcc compiler
• ❌ Not recommended for kernel compilation with NVIDIA drivers
• ✅ Sufficient for running pre-built binaries (if you skip Steps 2 and 7)

Prerequisites:

• Rocky Linux 9
• Root privileges
• Internet connectivity

Steps:

1. Complete installation script:

   # Install prerequisites
   dnf -y groupinstall "Development Tools"
   
   # Download and extract GCC 10 source
   cd /usr/local/src
   wget https://github.com/gcc-mirror/gcc/archive/refs/heads/releases/gcc-10.zip
   unzip gcc-10.zip
   cd gcc-releases-gcc-10
   
   # Download GCC prerequisites (GMP, MPFR, MPC, ISL)
   contrib/download_prerequisites
   
   # Create build directory and configure
   mkdir /usr/local/gcc-10
   cd /usr/local/gcc-10
   /usr/local/src/gcc-releases-gcc-10/configure --disable-multilib
   
   # Compile and install (1-2 hours depending on CPU)
   make -j $(nproc)
   make install
   

   Note

   : The compilation step make -j $(nproc) will take 1-2 hours depending on your CPU performance. The $(nproc) command uses all available CPU cores to speed up compilation.

2. Post-Install Configuration:

    # Configure dynamic linker to include both system and GCC 10 library paths
    cat > /etc/ld.so.conf.d/gcc-10.conf << 'EOF'
    /usr/lib64
    /usr/local/lib64
    EOF
   
    ldconfig
   
    # Update system compiler symlinks to use GCC 10
    rm -f /usr/bin/cc
    ln -s /usr/local/bin/gcc /usr/bin/cc
   

Verification:

# Verify GCC 10 installation
gcc --version
# Should output: gcc (GCC) 10.x.x

g++ --version
# Should output: g++ (GCC) 10.x.x

# Verify symlinks are correct
which cc
# Should output: /usr/bin/cc

ls -al /usr/bin/cc
# Should show: /usr/bin/cc -> /usr/local/bin/gcc

---

Step 2: Kernel Compilation (Required for NVIDIA 470 Compatibility)

Why Compile a Custom Kernel?

Recent kernel updates in Rocky Linux 9, Fedora, and Ubuntu have broken compatibility with:

• NVIDIA Driver 470 (required for Tesla K80 / Compute Capability 3.7)
• CUDA 11.4 nvcc compiler

Solution: Use Linux kernel 5.14.x, which maintains stable NVIDIA 470 driver support.

Prerequisites:

• Rocky Linux 9 (clean installation recommended)
• Root privileges
• At least 20GB free disk space
• Stable internet connection

Steps:

1. Install build tools:

   dnf -y groupinstall "Development Tools"
   dnf -y install ncurses-devel
   

2. Navigate to source directory:

   cd /usr/src/kernels
   

3. Download Linux 5.14.x kernel:

   wget https://www.kernel.org/pub/linux/kernel/v5.x/linux-5.14.tar.xz
   

   Note

   : Check kernel.org for the latest 5.14.x stable release.

4. Extract the archive:

   tar xvf linux-5.14.tar.xz
   cd linux-5.14
   

5. Copy existing kernel configuration:

   # Copy config from the running kernel
   cp /usr/src/kernels/$(uname -r)/.config .config
   

   Note

   : If you need to check available kernel configurations first: ls /usr/src/kernels

6. Open menuconfig to adjust settings:

   make menuconfig
   

7. Required Configuration Changes:

   Navigate and DISABLE the following options:

   a) Disable Module Signature Verification:

   Enable loadable module support
     → [ ] Module signature verification  (press N to disable)
   

   b) Disable Trusted Keys:

   Cryptographic API
     → Certificates for signature checking
       → [ ] Provide system-wide ring of trusted keys  (press N)
       → System trusted keys filename = "" (delete any content, leave empty)
   

   c) Disable BTF Debug Info:

   Kernel hacking
     → Compile-time checks and compiler options
       → [ ] Generate BTF typeinfo  (press N to disable CONFIG_DEBUG_INFO_BTF)
   

   Why disable these?

   • Module signatures: Prevents loading unsigned NVIDIA proprietary driver
   • Trusted keys: Conflicts with out-of-tree driver compilation
   • BTF debug: Can cause build failures and is unnecessary for production use

8. Save configuration:

   • Press <Save>
   • Confirm default filename .config
   • Press <Exit> to quit menuconfig

9. Compile kernel (using all CPU cores):

   make -j$(nproc)
   

   Estimated time: 30-60 minutes depending on CPU performance

10. Install kernel modules:

    make modules_install
    

11. Install kernel:

    make install
    

12. Reboot system:

    reboot
    

Verification:

# After reboot, verify kernel version
uname -r
# Should output: 5.14.21

Troubleshooting:

Issue: BTF-related build errors

BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
Failed to generate BTF for vmlinux

Solution:

• Disable CONFIG_DEBUG_INFO_BTF in menuconfig (see step 7c above)

Issue: Module signing key errors

Can't read private key

Solution:

• Disable CONFIG_MODULE_SIG_ALL and clear CONFIG_SYSTEM_TRUSTED_KEYS in menuconfig
• Ensure the "System trusted keys filename" field is completely empty

---

Step 3: NVIDIA Driver 470 Installation

Prerequisites:

• Rocky Linux 9 system running custom kernel 5.14.x (from Step 2)
• Root privileges
• Internet connectivity

Steps:

1. Update the system:

   dnf -y update
   

2. Install required dependencies:

   dnf -y install epel-release
   dnf -y install libglvnd-devel.x86_64
   

3. Switch to text mode (runlevel 3):

   init 3
   

   Note

   : This will exit the graphical interface. You'll need to log in via text console.

4. Download NVIDIA Driver 470.256.02:

   cd /tmp
   wget https://us.download.nvidia.com/tesla/470.256.02/NVIDIA-Linux-x86_64-470.256.02.run
   

5. Install NVIDIA Driver:

   chmod +x NVIDIA-Linux-x86_64-470.256.02.run
   sh NVIDIA-Linux-x86_64-470.256.02.run
   

   Installation prompts:

   • Accept the license agreement
   • If asked about DKMS, select "Yes" to register with DKMS
   • If asked about 32-bit compatibility libraries, select based on your needs
   • If asked about X configuration, select "Yes" if you use graphical interface

6. Reboot to load NVIDIA driver:

   reboot
   

Verification:

# Check driver and GPU
nvidia-smi
# Should show Tesla K80 GPU(s) with driver version 470.256.02

---

Step 4: CUDA 11.4 Toolkit Installation

Prerequisites:

• NVIDIA Driver 470 installed and verified (from Step 3)
• Root privileges

Steps:

1. Download CUDA 11.4.0 installer:

   cd /tmp
   wget https://developer.download.nvidia.com/compute/cuda/11.4.0/local_installers/cuda_11.4.0_470.42.01_linux.run
   

2. Run CUDA installer:

   sh cuda_11.4.0_470.42.01_linux.run
   

   Installation prompts:

   • Accept the license agreement
   • IMPORTANT: Deselect "Driver" option (driver already installed in Step 3)
   • Keep selected: CUDA Toolkit, CUDA Samples, CUDA Demo Suite, CUDA Documentation
   • Confirm installation

3. Set up CUDA Environment Variables:

   Create two configuration files:

   a) PATH configuration in /etc/profile.d/:

   cat > /etc/profile.d/cuda-11.4.sh << 'EOF'
   #!/bin/sh
   # cuda-11.4.sh - CUDA 11.4 PATH configuration for Tesla K80 support
   export PATH=/usr/local/cuda-11.4/bin${PATH:+:${PATH}}
   EOF
   
   # Apply PATH changes
   source /etc/profile.d/cuda-11.4.sh
   

   b) Dynamic linker configuration:

   The CUDA installer creates /etc/ld.so.conf.d/cuda-11-4.conf automatically with the following content:

   /usr/local/cuda-11.4/lib64
   /usr/local/cuda-11.4/targets/x86_64-linux/lib
   

   If the file doesn't exist or needs to be recreated:

   cat > /etc/ld.so.conf.d/cuda-11-4.conf << 'EOF'
   /usr/local/cuda-11.4/lib64
   /usr/local/cuda-11.4/targets/x86_64-linux/lib
   EOF
   
   # Update dynamic linker cache
   ldconfig
   

Verification:

# Check CUDA compiler
nvcc --version
# Should show: Cuda compilation tools, release 11.4, V11.4.48

# Check driver and CUDA compatibility
nvidia-smi
# Should show Tesla K80 GPU(s) with driver version 470.256.02 and CUDA Version: 11.4

---

Step 5: CMake 4.0 Installation

Prerequisites:

• Root privileges
• Internet connectivity

Steps:

1. Install OpenSSL Development Libraries:

   dnf -y install openssl-devel
   

2. Download CMake Source Code:

   cd /usr/local/src
   wget https://github.com/Kitware/CMake/releases/download/v4.0.0/cmake-4.0.0.tar.gz
   

3. Extract the Archive:

   tar xvf cmake-4.0.0.tar.gz
   

4. Create Installation Directory:

   mkdir /usr/local/cmake-4
   

5. Configure CMake:

   cd /usr/local/cmake-4
   /usr/local/src/cmake-4.0.0/configure
   

6. Compile CMake:

   make -j $(nproc)
   

7. Install CMake:

   make install
   

Verification:

cmake --version
# Should output: cmake version 4.0.0

---

Step 6: Go 1.25.3 Installation

Prerequisites:

• Root privileges
• Internet connectivity

Steps:

1. Download Go Distribution:

   cd /usr/local
   wget https://go.dev/dl/go1.25.3.linux-amd64.tar.gz
   

2. Extract the Archive:

   tar xvf go1.25.3.linux-amd64.tar.gz
   

3. Post Install Configuration:

   cat > /etc/profile.d/go.conf << 'EOF'
   #!/bin/sh
   # go.conf - Go environment configuration
   export PATH=/usr/local/go/bin${PATH:+:${PATH}}
   EOF
   
   # Apply the configuration
   source /etc/profile.d/go.conf
   

Verification:

go version
# Should output: go version go1.25.3 linux/amd64

---

Step 7: Ollama37 Compilation (Optional - For Custom Builds)

Prerequisites:

• GCC 10 (from Step 1)
• Rocky Linux 9 with custom kernel 5.14.x (from Step 2)
• NVIDIA Driver 470 (from Step 3)
• CUDA Toolkit 11.4 (from Step 4)
• CMake 4.0 (from Step 5)
• Go 1.25.3 (from Step 6)
• Git

Steps:

1. Navigate to Build Directory:

   cd /usr/local/src
   

2. Clone the Repository:

   git clone https://github.com/dogkeeper886/ollama37
   cd ollama37
   

3. CMake Configuration: Set compiler variables and configure the build system:

   CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -B build
   

4. CMake Build: Compile the C++ components (parallel build):

   CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake --build build -j$(nproc)
   

   Note: -j$(nproc) enables parallel compilation using all available CPU cores. You can specify a number like -j4 to limit the number of parallel jobs.

5. Go Build: Compile the Go components:

   go build -o ollama .
   

Verification:

./ollama --help

---

Build Verification

After successful compilation, verify Tesla K80 support:

# Check if ollama detects your GPU
./ollama serve &

# Pull a small model
./ollama pull llama3.2:3b

# Test inference
./ollama run llama3.2:3b "Hello Tesla K80!"

# Monitor GPU utilization
watch -n 1 nvidia-smi

---

Summary: Installation Paths

Path 1: Pre-built Binary (Easier)

1. ❌ Skip GCC 10 installation (not needed for pre-built binaries)
2. Compile custom kernel 5.14.x
3. Install NVIDIA Driver 470
4. Install CUDA 11.4 Toolkit
5. Install CMake 4.0
6. Install Go 1.25.3
7. Download and run pre-built ollama37 binary

Path 2: Compile from Source (Advanced - Requires All Steps)

1. ✅ Install GCC 10 (required for kernel and ollama37 compilation)
2. Compile custom kernel 5.14.x (uses GCC 10)
3. Install NVIDIA Driver 470
4. Install CUDA 11.4 Toolkit
5. Install CMake 4.0
6. Install Go 1.25.3
7. Compile ollama37 from source (uses GCC 10)

Choose the path that best fits your requirements and technical expertise.