ollama37/llm/CLAUDE.md

# LLM Package - Memory Estimation Optimization Guide

**Status**: ⚠️ **IN PROGRESS** - Implementation pending

This file contains instructions for optimizing GPU memory estimation to reduce unnecessary multi-GPU splits on Tesla K80 dual-GPU systems.

---

## 🎯 Goal

Fix the graph memory overestimation that causes models to split across multiple GPUs when they could fit on a single GPU.

**Objective**: Reduce total estimated memory from 13.0 GiB → 10.8 GiB for gemma3:12b, allowing it to run on a single Tesla K80 (11.2 GiB).

---

## 📊 Problem Analysis (2025-10-29)

### Real-World Measurements

**Test Case**: `gemma3:12b` model on dual Tesla K80 GPUs

**Current Behavior**:
```
Estimated Memory:
  GPU 0: 7.7 GiB  (graph: 1.3 GiB allocated)
  GPU 1: 5.3 GiB  (graph: 1.3 GiB allocated)
  Total: 13.0 GiB (graph: 2.6 GiB total)

Actual Memory (nvidia-smi):
  GPU 0: 4.1 GiB  (graph: 181.3 MiB actual)
  GPU 1: 6.3 GiB  (graph: 1.1 GiB actual)
  Total: 10.4 GiB (graph: 1.28 GiB total)

Result: Split across 2 GPUs (slower inference)
```

### Root Cause

**File**: `llm/memory.go`
**Lines**: 289-298
**Issue**: Graph memory is allocated to ALL GPUs at 100%, but only the primary GPU needs full graph size.

```go
// Current problematic code:
for i := range gpus {
    if layerCounts[i] <= 0 {
        continue
    }
    if fullyLoaded {
        gpuAllocations[i] += graphFullOffload  // ← 1.3 GiB added to EACH GPU
    } else {
        gpuAllocations[i] += graphPartialOffload  // ← 1.3 GiB added to EACH GPU
    }
}
```

### Empirical Findings

From actual Tesla K80 measurements:
- **Primary GPU** (GPU 1 - most layers): Needs full graph (1.1 GiB ≈ 100% of estimate)
- **Secondary GPU** (GPU 0 - fewer layers): Needs minimal graph (181.3 MiB ≈ 14% of estimate)
- **Ratio**: Secondary GPU uses ~1/7 (14%) of estimated graph size

---

## 🔧 Implementation Instructions

### Step 1: Locate the Target Code

**File**: `/home/jack/Documents/ollama37/llm/memory.go`
**Target Lines**: 289-298

Original code block:
```go
// Add the applicable (full or partial) graph allocations
for i := range gpus {
    if layerCounts[i] <= 0 {
        continue
    }
    if fullyLoaded {
        gpuAllocations[i] += graphFullOffload
    } else {
        gpuAllocations[i] += graphPartialOffload
    }
}
```

### Step 2: Replace with Optimized Code

**Action**: Replace lines 289-298 with the following:

```go
// Add the applicable (full or partial) graph allocations
// ollama37: Multi-GPU optimization for Tesla K80
// Primary GPU (last GPU with most layers) needs full graph memory
// Secondary GPUs only need ~15% of graph size based on empirical measurements
for i := range gpus {
    if layerCounts[i] <= 0 {
        continue
    }

    var graphAlloc uint64

    // Determine which GPU gets full graph vs reduced graph
    if len(gpus) > 1 && i < len(gpus)-1 {
        // Secondary GPU: Use 15% of graph size
        // Empirical data: GPU 0 used 181.3 MiB vs 1.3 GiB estimate = ~14% ratio
        // Using 1/7 ratio (14.3%) provides conservative buffer
        if fullyLoaded {
            graphAlloc = graphFullOffload / 7
        } else {
            graphAlloc = graphPartialOffload / 7
        }
    } else {
        // Primary GPU (or single GPU): Full graph allocation
        if fullyLoaded {
            graphAlloc = graphFullOffload
        } else {
            graphAlloc = graphPartialOffload
        }
    }

    gpuAllocations[i] += graphAlloc
}
```

### Step 3: Verification

After making the change, verify the code compiles:

```bash
# Navigate to project root
cd /home/jack/Documents/ollama37

# Build Go binary
go build -o ollama .

# Should complete without errors
echo $?  # Should output: 0
```

---

## 🧪 Testing Procedure

### Test 1: Memory Estimation Check

**Objective**: Verify the estimated memory now fits in single GPU

```bash
# Start ollama server
./ollama serve &

# Wait for server to start
sleep 2

# Load gemma3:12b and watch logs
./ollama run gemma3:12b

# Expected log output should show:
# memory.required.allocations="[X.X GiB Y.Y GiB]"
# Where total (X.X + Y.Y) ≈ 10.8 GiB (down from 13.0 GiB)
#
# With the fix:
# - GPU 0: ~5.5 GiB (down from 7.7 GiB) = 7.7 - 1.3 + (1.3/7) = 5.5 GiB
# - GPU 1: ~5.3 GiB (unchanged)
# - Total: ~10.8 GiB (down from 13.0 GiB)
```

### Test 2: Single GPU Loading

**Objective**: Verify model now loads on single GPU instead of splitting

```bash
# Monitor GPU memory during load
watch -n 1 nvidia-smi

# Expected behavior:
# BEFORE FIX: Model splits across GPU 0 (4.1 GiB) + GPU 1 (6.3 GiB)
# AFTER FIX:  Model loads on single GPU (likely GPU 1: ~10.4 GiB)
```

### Test 3: Inference Performance

**Objective**: Verify inference still works correctly

```bash
# Run inference test
./ollama run gemma3:12b "Explain quantum computing in one sentence."

# Expected:
# - Response should generate successfully
# - Check nvidia-smi during inference
# - Verify GPU utilization is normal (>80%)
```

### Test 4: Multi-GPU Models Still Work

**Objective**: Ensure models that TRULY need multi-GPU still split correctly

```bash
# Test with a larger model that requires >11 GiB
# (If you have one available)
./ollama run [larger-model]

# Expected:
# - Should still split across both GPUs
# - Primary GPU should still get full graph
# - Secondary GPUs should still get reduced graph
```

---

## 📈 Expected Results

### Memory Allocation Improvements

| Metric | Before Fix | After Fix | Improvement |
|--------|-----------|-----------|-------------|
| **GPU 0 allocation** | 7.7 GiB | 5.5 GiB | -2.2 GiB (29%) |
| **GPU 1 allocation** | 5.3 GiB | 5.3 GiB | unchanged |
| **Total estimate** | 13.0 GiB | 10.8 GiB | -2.2 GiB (17%) |
| **Fits single K80?** | ❌ No | ✅ Yes | ✓ |
| **Graph overhead** | 2.6 GiB | 1.48 GiB | -1.12 GiB (43%) |

### Performance Expectations

**Single-GPU Mode (NEW)**:
- ✅ Faster inference (no cross-GPU communication)
- ✅ Simpler memory management
- ✅ Better VRAM utilization
- ✅ Models up to ~10.5 GiB can run on single GPU

**Multi-GPU Mode (for larger models)**:
- ✅ Still works correctly for models >11 GiB
- ✅ More accurate memory estimation
- ✅ Reduced wasted VRAM on secondary GPUs

---

## 🔍 How It Works

### Memory Allocation Logic

**Key Insight**: In multi-GPU splits, the **last GPU (highest index)** typically has the most layers and handles the output layer, requiring full graph memory. Earlier GPUs handle fewer intermediate layers and need minimal graph memory.

**Layer Distribution Example** (gemma3:12b):
```
layers.split="25,24"
  GPU 0: 25 layers (intermediate layers only)
  GPU 1: 24 layers (includes output layer)

GPU 1 needs full graph for final computations
GPU 0 only needs small graph for intermediate passes
```

### Graph Memory Breakdown

```
Full Graph Memory: 1.3 GiB (from model.graphFullOffload)

Multi-GPU Allocation:
  GPU 0 (secondary): 1.3 GiB / 7 = 186 MiB  (~14% - matches empirical 181.3 MiB)
  GPU 1 (primary):   1.3 GiB / 1 = 1.3 GiB  (100% - matches empirical 1.1 GiB)

Total Graph: 1.49 GiB (vs 2.6 GiB before) = 43% reduction
```

### Ratio Selection Rationale

**Why 1/7 (14.3%)?**

1. **Empirical measurement**: GPU 0 used 181.3 MiB / 1.3 GiB = 14.0%
2. **Conservative buffer**: 1/7 = 14.3% provides slight headroom
3. **Simple integer division**: Easy to compute, no floating point
4. **Validated**: Matches real-world usage within 3%

**Alternative ratios to consider** (if needed):
- `/ 8` = 12.5% (more aggressive)
- `/ 6` = 16.7% (more conservative)
- `/ 5` = 20.0% (very conservative)

Current choice of `/7` provides best balance of accuracy and safety margin.

---

## 🐛 Troubleshooting

### Issue: Model still splits across GPUs after fix

**Diagnosis**:
```bash
# Check the log output for memory.required.allocations
grep "memory.required.allocations" /path/to/ollama.log
```

**Possible causes**:
1. Code change not applied correctly - verify lines 289-298
2. Binary not rebuilt - run `go build -o ollama .` again
3. Old process still running - `pkill ollama` and restart

### Issue: Compilation errors after change

**Error**: `undefined: graphAlloc`

**Solution**: Ensure the entire for-loop block (lines 289-298) is replaced, not just part of it.

### Issue: Out of memory errors during inference

**Symptoms**: Model loads but fails during generation

**Solution**: The 1/7 ratio may be too aggressive. Edit memory.go and change:
```go
graphAlloc = graphFullOffload / 7  // Change to /6 for more headroom
```

### Issue: Model loads but inference is slow

**Diagnosis**: Check if model actually loaded on single GPU:
```bash
nvidia-smi  # During inference
```

**Expected**: One GPU should show ~10-11 GiB usage, other GPU minimal
**If both GPUs active**: Model may still be splitting (check logs)

---

## 📝 Additional Notes

### Preserving Multi-GPU Functionality

This optimization ONLY affects multi-GPU systems. Single-GPU systems are unaffected because:

```go
if len(gpus) > 1 && i < len(gpus)-1 {
    // Only executes when multiple GPUs present
}
```

### Future Enhancements (Optional)

If more fine-tuning is needed, consider:

1. **Model-specific ratios**: Larger models might need different ratios
2. **Layer-count based calculation**: Scale ratio based on layer distribution
3. **Environment variable**: `OLLAMA_SECONDARY_GPU_GRAPH_RATIO` for user control

For now, the hardcoded 1/7 ratio provides best results for Tesla K80 based on empirical data.

---

## ✅ Completion Checklist

- [ ] **Code modified**: Lines 289-298 in `llm/memory.go` replaced with optimized version
- [ ] **Build successful**: `go build -o ollama .` completes without errors
- [ ] **Test 1 passed**: Memory estimation reduced to ~10.8 GiB
- [ ] **Test 2 passed**: Model loads on single GPU
- [ ] **Test 3 passed**: Inference works correctly
- [ ] **Test 4 passed**: Large models still split when needed
- [ ] **Documentation updated**: Update root CLAUDE.md status from "IN PROGRESS" to "COMPLETED"
- [ ] **Performance verified**: Single-GPU inference faster than multi-GPU split

Once all items checked, update status in `/home/jack/Documents/ollama37/CLAUDE.md`:
```markdown
**Status**: ✅ **COMPLETED** - Single-GPU preference optimization deployed
```

---

## 📚 Reference

**Related Files**:
- `llm/memory.go` - Memory estimation logic (THIS FILE)
- `llm/server.go` - LLM server process management
- `server/sched.go` - GPU scheduler
- `discover/gpu.go` - GPU detection and capabilities

**Key Functions**:
- `EstimateGPULayers()` - Main memory estimation function (line 74)
- `PredictServerFit()` - Determines if model fits in available VRAM (line 18)

**Empirical Data Source**:
- User logs from 2025-10-29 showing gemma3:12b memory usage
- nvidia-smi measurements during actual inference
- Ollama server logs with detailed memory allocations