matt/ollama37
1
0
Fork 0
mirror of https://github.com/dogkeeper886/ollama37.git synced 2025-12-10 07:46:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
d39e39e6124d949edafe951ef9cb4dd78090b60c
ollama37/README.md
Shang Chieh Tseng d39e39e612 Update README.md
2025-04-19 10:04:35 +08:00

45 KiB
Raw Blame History

  ollama

Ollama

Installation Guide: CUDA 11.4 on Rocky Linux 8

Prerequisites:

  • A Rocky Linux 8 system or a container based on Rocky Linux 8.
  • Root privileges.
  • Internet connectivity.

Steps:

  1. Update the system: Start by updating the operating system packages.

    dnf -y update

  2. Install EPEL Repository: The Extra Packages for Enterprise Linux (EPEL) repository is required for some dependencies.

    dnf -y install epel-release

  3. Add NVIDIA CUDA Repository: Add the NVIDIA CUDA repository for RHEL 8. This allows dnf to find and install the necessary CUDA packages.

    dnf -y config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

  4. Install NVIDIA Driver (Version 470): Install the NVIDIA driver version 470 using the DKMS (Dynamic Kernel Module Support) system. This ensures the driver is automatically rebuilt when the kernel is updated.

    dnf -y module install nvidia-driver:470-dkms

  5. Install CUDA Toolkit 11.4: Install the CUDA Toolkit version 11.4.

    dnf -y install cuda-11-4

  6. Set up CUDA Environment Variables (Optional but Recommended): Copy a script (cuda-11.4.sh) to /etc/profile.d/ to set environment variables. If performing this installation manually, you need to ensure that the PATH and LD_LIBRARY_PATH environment variables are correctly configured. The contents of cuda-11.4.sh would typically include something like:

    # Create /etc/profile.d/cuda-11.4.sh
echo "export PATH=/usr/local/cuda-11.4/bin:${PATH}" > /etc/profile.d/cuda-11.4.sh
echo "export LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64:${LD_LIBRARY_PATH}" >> /etc/profile.d/cuda-11.4.sh

    To apply these changes, you can either:

    • Source the script: source /etc/profile.d/cuda-11.4.sh
    • Log out and log back in to refresh your shell environment.
    • Add the lines above to your .bashrc or equivalent shell configuration file.

Verification:

After installation, verify that CUDA is properly installed by running:

nvcc --version

This command should display the CUDA compiler version. You can also check the installed driver version with:

nvidia-smi

GCC 10 Installation Guide

This guide details the steps to install GCC 10.

Steps:

  1. Update and Install Prerequisites:

    • Installs wget, unzip, and lbzip2 to download and extract the GCC source code.
    • Installs the "Development Tools" group, which includes necessary build tools.
    dnf -y install wget unzip lbzip2 \
    && dnf -y groupinstall "Development Tools"

  2. Download GCC 10 Source Code:

    • Downloads the GCC 10 source code from the gcc-mirror GitHub repository.
    cd /usr/local/src \
    && wget https://github.com/gcc-mirror/gcc/archive/refs/heads/releases/gcc-10.zip

  3. Extract Source Code:

    • Unzips the downloaded GCC 10 archive.
    unzip gcc-10.zip

  4. Prepare Build Environment:

    • Navigates into the extracted GCC 10 directory.
    cd gcc-releases-gcc-10

  5. Download Prerequisites:

    • Downloads build prerequisites using the contrib/download_prerequisites script.
    contrib/download_prerequisites

  6. Create Installation Directory:

    • Creates a directory /usr/local/gcc-10 where GCC 10 will be installed.
    mkdir /usr/local/gcc-10

  7. Configure GCC Build:

    • Configures the GCC build process using the ./configure script. The --disable-multilib flag disables the build of multilib support, which can simplify the build.
    cd /usr/local/gcc-10 \
&& /usr/local/src/gcc-releases-gcc-10/configure --disable-multilib

  8. Compile GCC:

    • Compiles GCC using make. The -j ${nproc} flag utilizes all available CPU cores for parallel compilation, speeding up the process.
    make -j ${nproc}

  9. Install GCC:

    • Installs the compiled GCC binaries.
    make install

  10. Post-Install Configuration:

    • Configures the system environment for GCC 10 compatibility. This involves creating a file (/etc/profile.d/gcc-10.sh) to automatically set LD_LIBRARY_PATH="/usr/local/lib64" and adding a configuration file (/etc/ld.so.conf.d/gcc-10.conf) to update the dynamic linker's cache.
    # Create /etc/profile.d/gcc-10.sh
echo "export LD_LIBRARY_PATH=/usr/local/lib64:\$LD_LIBRARY_PATH" > /etc/profile.d/gcc-10.sh

# Create /etc/ld.so.conf.d/gcc-10.conf
echo "/usr/local/lib64" > /etc/ld.so.conf.d/gcc-10.conf

# Update dynamic linker cache
ldconfig

CMake Installation Guide

  1. Update Package Manager (Optional but Recommended):

    While not explicitly in the Dockerfile, it's good practice to start by updating your package lists to ensure you're getting the latest available software. This isn't shown in the provided Dockerfile.

  2. Install OpenSSL Development Libraries:

    dnf -y install openssl-devel
    • Purpose: CMake often relies on OpenSSL for secure build configurations. The -devel package provides the header files and libraries necessary for building software that uses OpenSSL.
    • dnf: This is the package manager for Fedora and related distributions (like CentOS, Rocky Linux, etc.). If you're using a different distribution, use the appropriate package manager (e.g., apt for Debian/Ubuntu, yum for older CentOS versions).
    • -y: This flag automatically answers "yes" to any prompts during the installation, making the process non-interactive.

  3. Download CMake Source Code:

    cd /usr/local/src
wget https://github.com/Kitware/CMake/releases/download/v4.0.0/cmake-4.0.0.tar.gz
    • cd /usr/local/src: Changes the current directory to /usr/local/src. This is a common location for temporary source files.
    • wget: Downloads the CMake source code archive from the specified URL. wget is a command-line utility for retrieving files from the web. Make sure you have wget installed.

  4. Extract the Archive:

    tar xvf cmake-4.0.0.tar.gz
    • tar: This is the GNU Tape Archiver, a common utility for creating and extracting archive files.
    • xvf: These are the tar options:
      • x: Extract files.
      • v: Verbose mode (lists the files being extracted).
      • f : Specifies the archive file.

  5. Create a CMake Installation Directory:

    mkdir /usr/local/cmake-4
    • mkdir: Creates a new directory. This directory is where CMake will be installed.

  6. Configure CMake:

    cd /usr/local/cmake-4
/usr/local/src/cmake-4.0.0/configure
    • cd /usr/local/cmake-4: Changes the directory to the newly created installation directory.
    • /usr/local/src/cmake-4.0.0/configure: This script prepares the CMake source code for compilation based on your system's configuration. It checks for dependencies and creates Makefiles.

  7. Compile CMake:

    make -j ${nproc}
    • make: This command compiles the CMake source code.
    • -j ${nproc}: This option tells make to use multiple processor cores to speed up the compilation. ${nproc} is an environment variable that contains the number of available processors.

  8. Install CMake:

    make install
    • make install: This command installs the compiled CMake binaries and related files to the system directories. This step usually requires root privileges (e.g., using sudo).

Go Installation Guide

This guide installs Go version 1.24.2, as specified in the Dockerfile.

  1. Download Go Distribution:

    cd /usr/local
wget https://go.dev/dl/go1.24.2.linux-amd64.tar.gz
    • cd /usr/local: Changes the current directory to /usr/local. This is a common location for installing software.
    • wget: Downloads the Go distribution archive from the specified URL. Ensure that wget is installed on your system.

  2. Extract the Archive:

    tar xvf go1.24.2.linux-amd64.tar.gz
    • tar: The GNU Tape Archiver.
    • xvf: As described in the CMake installation guide, these options extract the archive and list the extracted files.

Compilation Guide: Ollama37

Prerequisites:

  • Rocky Linux 8.
  • git: For cloning the repository.
  • cmake: For managing the C++ build process.
  • go: The Go compiler and toolchain.
  • gcc: version 10 (GNU Compiler Collection) and G++: For C++ compilation (although the Dockerfile explicitly sets these via CC and CXX)
  • CUDA Toolkit 11.4

Steps:

  1. Navigate to the Build Directory:

    cd /usr/local/src

  2. Clone the Repository:

    git clone https://github.com/dogkeeper886/ollama37

  3. Change Directory:

    cd ollama37

  4. CMake Configuration:

    This step configures the build system. The CC and CXX variables are explicitly set to /usr/local/bin/gcc and /usr/local/bin/g++, respectively. This is critical if the system's default compilers are incompatible or need to be overridden.

    CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -B build

  5. CMake Build:

    This step actually compiles the C++ code using the configuration generated in the previous step.

    CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake --build build

  6. Go Build:

    Finally, this step compiles the Go code, creating the ollama executable.

    go build -o ollama .

macOS

Download

Windows

Download

Linux

curl -fsSL https://ollama.com/install.sh | sh

Manual install instructions

Docker

The official Ollama Docker image ollama/ollama is available on Docker Hub.

Libraries

Community

Quickstart

To run and chat with Llama 3.2:

ollama run llama3.2

Model library

Ollama supports a list of models available on ollama.com/library

Here are some example models that can be downloaded:

Model Parameters Size Download
Gemma 3 1B 815MB ollama run gemma3:1b
Gemma 3 4B 3.3GB ollama run gemma3
Gemma 3 12B 8.1GB ollama run gemma3:12b
Gemma 3 27B 17GB ollama run gemma3:27b
QwQ 32B 20GB ollama run qwq
DeepSeek-R1 7B 4.7GB ollama run deepseek-r1
DeepSeek-R1 671B 404GB ollama run deepseek-r1:671b
Llama 3.3 70B 43GB ollama run llama3.3
Llama 3.2 3B 2.0GB ollama run llama3.2
Llama 3.2 1B 1.3GB ollama run llama3.2:1b
Llama 3.2 Vision 11B 7.9GB ollama run llama3.2-vision
Llama 3.2 Vision 90B 55GB ollama run llama3.2-vision:90b
Llama 3.1 8B 4.7GB ollama run llama3.1
Llama 3.1 405B 231GB ollama run llama3.1:405b
Phi 4 14B 9.1GB ollama run phi4
Phi 4 Mini 3.8B 2.5GB ollama run phi4-mini
Mistral 7B 4.1GB ollama run mistral
Moondream 2 1.4B 829MB ollama run moondream
Neural Chat 7B 4.1GB ollama run neural-chat
Starling 7B 4.1GB ollama run starling-lm
Code Llama 7B 3.8GB ollama run codellama
Llama 2 Uncensored 7B 3.8GB ollama run llama2-uncensored
LLaVA 7B 4.5GB ollama run llava
Granite-3.2 8B 4.9GB ollama run granite3.2

Note

You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.

Customize a model

Import from GGUF

Ollama supports importing GGUF models in the Modelfile:

  1. Create a file named Modelfile, with a FROM instruction with the local filepath to the model you want to import.

    FROM ./vicuna-33b.Q4_0.gguf

  2. Create the model in Ollama

    ollama create example -f Modelfile

  3. Run the model

    ollama run example

Import from Safetensors

See the guide on importing models for more information.

Customize a prompt

Models from the Ollama library can be customized with a prompt. For example, to customize the llama3.2 model:

ollama pull llama3.2

Create a Modelfile:

FROM llama3.2

# set the temperature to 1 [higher is more creative, lower is more coherent]
PARAMETER temperature 1

# set the system message
SYSTEM """
You are Mario from Super Mario Bros. Answer as Mario, the assistant, only.
"""

Next, create and run the model:

ollama create mario -f ./Modelfile
ollama run mario
>>> hi
Hello! It's your friend Mario.

For more information on working with a Modelfile, see the Modelfile documentation.

CLI Reference

Create a model

ollama create is used to create a model from a Modelfile.

ollama create mymodel -f ./Modelfile

Pull a model

ollama pull llama3.2

This command can also be used to update a local model. Only the diff will be pulled.

Remove a model

ollama rm llama3.2

Copy a model

ollama cp llama3.2 my-model

Multiline input

For multiline input, you can wrap text with """:

>>> """Hello,
... world!
... """
I'm a basic program that prints the famous "Hello, world!" message to the console.

Multimodal models

ollama run llava "What's in this image? /Users/jmorgan/Desktop/smile.png"

Output: The image features a yellow smiley face, which is likely the central focus of the picture.

Pass the prompt as an argument

ollama run llama3.2 "Summarize this file: $(cat README.md)"

Output: Ollama is a lightweight, extensible framework for building and running language models on the local machine. It provides a simple API for creating, running, and managing models, as well as a library of pre-built models that can be easily used in a variety of applications.

Show model information

ollama show llama3.2

List models on your computer

ollama list

List which models are currently loaded

ollama ps

Stop a model which is currently running

ollama stop llama3.2

Start Ollama

ollama serve is used when you want to start ollama without running the desktop application.

Building

See the developer guide

Running local builds

Next, start the server:

./ollama serve

Finally, in a separate shell, run a model:

./ollama run llama3.2

REST API

Ollama has a REST API for running and managing models.

Generate a response

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt":"Why is the sky blue?"
}'

Chat with a model

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "why is the sky blue?" }
  ]
}'

See the API documentation for all endpoints.

Community Integrations

Web & Desktop

Cloud

Terminal

Apple Vision Pro

  • SwiftChat (Cross-platform AI chat app supporting Apple Vision Pro via "Designed for iPad")
  • Enchanted

Database

  • pgai - PostgreSQL as a vector database (Create and search embeddings from Ollama models using pgvector)
  • MindsDB (Connects Ollama models with nearly 200 data platforms and apps)
  • chromem-go with example
  • Kangaroo (AI-powered SQL client and admin tool for popular databases)

Package managers

Libraries

Mobile

  • SwiftChat (Lightning-fast Cross-platform AI chat app with native UI for Android, iOS and iPad)
  • Enchanted
  • Maid
  • Ollama App (Modern and easy-to-use multi-platform client for Ollama)
  • ConfiChat (Lightweight, standalone, multi-platform, and privacy focused LLM chat interface with optional encryption)
  • Ollama Android Chat (No need for Termux, start the Ollama service with one click on an Android device)
  • Reins (Easily tweak parameters, customize system prompts per chat, and enhance your AI experiments with reasoning model support.)

Extensions & Plugins

Supported backends

  • llama.cpp project founded by Georgi Gerganov.

Observability

  • Opik is an open-source platform to debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards. Opik supports native intergration to Ollama.
  • Lunary is the leading open-source LLM observability platform. It provides a variety of enterprise-grade features such as real-time analytics, prompt templates management, PII masking, and comprehensive agent tracing.
  • OpenLIT is an OpenTelemetry-native tool for monitoring Ollama Applications & GPUs using traces and metrics.
  • HoneyHive is an AI observability and evaluation platform for AI agents. Use HoneyHive to evaluate agent performance, interrogate failures, and monitor quality in production.
  • Langfuse is an open source LLM observability platform that enables teams to collaboratively monitor, evaluate and debug AI applications.
  • MLflow Tracing is an open source LLM observability tool with a convenient API to log and visualize traces, making it easy to debug and evaluate GenAI applications.
Reference in New Issue View Git Blame Copy Permalink