matt/ollama37
1
0
Fork 0
mirror of https://github.com/dogkeeper886/ollama37.git synced 2025-12-10 15:57:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Split Tesla K80 workflows into build and test; add test framework plan

Browse Source 
- Changed tesla-k80-ci.yml to manual trigger only, simplified to build-only workflow
- Created tesla-k80-tests.yml for separate test execution (manual trigger)
- Added .github/workflows/CLAUDE.md with comprehensive test framework design
- Removed binary artifact upload (not needed for single self-hosted runner)
- Replaced README.md with CLAUDE.md for better documentation structure

Test framework plan:
- Go-based test runner at cmd/test-runner/
- YAML configuration for multi-model testing
- Server lifecycle management with log monitoring
- API-based testing with structured reporting
- Support for test profiles (quick/full/stress)
This commit is contained in:
Shang Chieh Tseng
2025-10-30 10:59:52 +08:00
parent 7e317fdd74
commit b402b073c5
4 changed files with 420 additions and 299 deletions
Download Patch File Download Diff File Expand all files Collapse all files

266
.github/workflows/CLAUDE.md vendored Normal file
View File

@@ -0,0 +1,266 @@
# GitHub Actions Workflows - Tesla K80 Testing
## Overview
This directory contains workflows for automated testing of ollama37 on Tesla K80 (CUDA Compute Capability 3.7) hardware.
## Workflows
### 1. tesla-k80-ci.yml - Build Workflow
**Trigger**: Manual only (`workflow_dispatch`)
**Purpose**: Build the ollama binary with CUDA 3.7 support
**Steps**:
1. Checkout code
2. Clean previous build artifacts
3. Configure CMake with GCC 10 and CUDA 11
4. Build C++/CUDA components
5. Build Go binary
6. Verify binary
7. Upload binary artifact
**Artifacts**: `ollama-binary-{sha}` - Compiled binary for the commit
### 2. tesla-k80-tests.yml - Test Workflow
**Trigger**: Manual only (`workflow_dispatch`)
**Purpose**: Run comprehensive tests using the test framework
**Steps**:
1. Checkout code
2. Verify ollama binary exists
3. Run test-runner tool (see below)
4. Upload test results and logs
**Artifacts**: Test reports, logs, analysis results
## Test Framework Architecture
### TODO: Implement Go-based Test Runner
**Goal**: Create a dedicated Go test orchestrator at `cmd/test-runner/main.go` that manages the complete test lifecycle for Tesla K80.
#### Task Breakdown
1. **Design test configuration format**
- Create `test/config/models.yaml` - List of models to test with parameters
- Define model test spec: name, size, expected behavior, test prompts
- Support test profiles: quick (small models), full (all sizes), stress test
2. **Implement server lifecycle management**
- Start `./ollama serve` as subprocess
- Capture stdout/stderr to log file
- Monitor server readiness (health check API)
- Graceful shutdown on test completion or failure
- Timeout handling for hung processes
3. **Implement real-time log monitoring**
- Goroutine to tail server logs
- Pattern matching for critical events:
- GPU initialization messages
- Model loading progress
- CUDA errors or warnings
- Memory allocation failures
- CPU fallback warnings
- Store events for later analysis
4. **Implement model testing logic**
- For each model in config:
- Pull model via API (if not cached)
- Wait for model ready
- Parse logs for GPU loading confirmation
- Send chat API request with test prompt
- Validate response (not empty, reasonable length, coherent)
- Check logs for errors during inference
- Record timing metrics (load time, first token, completion)
5. **Implement test validation**
- GPU loading verification:
- Parse logs for "loaded model" + GPU device ID
- Check for "offloading N layers to GPU"
- Verify no "using CPU backend" messages
- Response quality checks:
- Response not empty
- Minimum token count (avoid truncated responses)
- JSON structure valid (for API responses)
- Error detection:
- No CUDA errors in logs
- No OOM (out of memory) errors
- No model loading failures
6. **Implement structured reporting**
- Generate JSON report with:
- Test summary (pass/fail/skip counts)
- Per-model results (status, timings, errors)
- Log excerpts for failures
- GPU metrics (memory usage, utilization)
- Generate human-readable summary (markdown/text)
- Exit code: 0 for all pass, 1 for any failure
7. **Implement CLI interface**
- Flags:
- `--config` - Path to test config file
- `--profile` - Test profile to run (quick/full/stress)
- `--ollama-bin` - Path to ollama binary (default: ./ollama)
- `--output` - Report output path
- `--verbose` - Detailed logging
- `--keep-models` - Don't delete models after test
- Subcommands:
- `run` - Run tests
- `validate` - Validate config only
- `list` - List available test profiles/models
8. **Update GitHub Actions workflow**
- Build test-runner binary in CI workflow
- Run test-runner in test workflow
- Parse JSON report for pass/fail
- Upload structured results as artifacts
#### File Structure
```
cmd/test-runner/
main.go # CLI entry point
config.go # Config loading and validation
server.go # Server lifecycle management
monitor.go # Log monitoring and parsing
test.go # Model test execution
validate.go # Response and log validation
report.go # Test report generation
test/config/
models.yaml # Default test configuration
quick.yaml # Quick test profile (small models)
full.yaml # Full test profile (all sizes)
.github/workflows/
tesla-k80-ci.yml # Build workflow (manual)
tesla-k80-tests.yml # Test workflow (manual, uses test-runner)
```
#### Example Test Configuration (models.yaml)
```yaml
profiles:
quick:
models:
- name: gemma2:2b
prompts:
- "Hello, respond with a greeting."
min_response_tokens: 5
timeout: 30s
full:
models:
- name: gemma2:2b
prompts:
- "Hello, respond with a greeting."
- "What is 2+2?"
min_response_tokens: 5
timeout: 30s
- name: gemma3:4b
prompts:
- "Explain photosynthesis in one sentence."
min_response_tokens: 10
timeout: 60s
- name: gemma3:12b
prompts:
- "Write a haiku about GPUs."
min_response_tokens: 15
timeout: 120s
validation:
gpu_required: true
check_patterns:
success:
- "loaded model"
- "offload.*GPU"
failure:
- "CUDA.*error"
- "out of memory"
- "CPU backend"
```
#### Example Test Runner Usage
```bash
# Build test runner
go build -o test-runner ./cmd/test-runner
# Run quick test profile
./test-runner run --config test/config/models.yaml --profile quick
# Run full test with verbose output
./test-runner run --profile full --verbose --output test-report.json
# Validate config only
./test-runner validate --config test/config/models.yaml
# List available profiles
./test-runner list
```
#### Integration with GitHub Actions
```yaml
- name: Build test runner
run: go build -o test-runner ./cmd/test-runner
- name: Run tests
run: |
./test-runner run --profile full --output test-report.json --verbose
timeout-minutes: 45
- name: Check test results
run: |
if ! jq -e '.summary.failed == 0' test-report.json; then
echo "Tests failed!"
jq '.failures' test-report.json
exit 1
fi
- name: Upload test report
uses: actions/upload-artifact@v4
with:
name: test-report
path: |
test-report.json
ollama.log
```
## Prerequisites
### Self-Hosted Runner Setup
1. **Install GitHub Actions Runner on your Tesla K80 machine**:
```bash
mkdir -p ~/actions-runner && cd ~/actions-runner
curl -o actions-runner-linux-x64-2.XXX.X.tar.gz -L \
https://github.com/actions/runner/releases/download/vX.XXX.X/actions-runner-linux-x64-2.XXX.X.tar.gz
tar xzf ./actions-runner-linux-x64-2.XXX.X.tar.gz
# Configure (use token from GitHub)
./config.sh --url https://github.com/YOUR_USERNAME/ollama37 --token YOUR_TOKEN
# Install and start as a service
sudo ./svc.sh install
sudo ./svc.sh start
```
2. **Verify runner environment has**:
- CUDA 11.4+ toolkit installed
- GCC 10 at `/usr/local/bin/gcc` and `/usr/local/bin/g++`
- CMake 3.24+
- Go 1.24+
- NVIDIA driver with Tesla K80 support
- Network access to download models
## Security Considerations
- Self-hosted runners should be on a secure, isolated machine
- Consider using runner groups to restrict repository access
- Do not use self-hosted runners for public repositories (untrusted PRs)
- Keep runner software updated

150
.github/workflows/README.md vendored
View File

@@ -1,150 +0,0 @@
# GitHub Actions Workflows
## Tesla K80 CI Workflow
The `tesla-k80-ci.yml` workflow builds and tests ollama with CUDA Compute Capability 3.7 support using a self-hosted runner.
### Prerequisites
#### Self-Hosted Runner Setup
1. **Install GitHub Actions Runner on your Tesla K80 machine**:
```bash
# Navigate to your repository on GitHub:
# Settings > Actions > Runners > New self-hosted runner
# Follow the provided instructions to download and configure the runner
mkdir -p ~/actions-runner && cd ~/actions-runner
curl -o actions-runner-linux-x64-2.XXX.X.tar.gz -L \
https://github.com/actions/runner/releases/download/vX.XXX.X/actions-runner-linux-x64-2.XXX.X.tar.gz
tar xzf ./actions-runner-linux-x64-2.XXX.X.tar.gz
# Configure (use token from GitHub)
./config.sh --url https://github.com/YOUR_USERNAME/ollama37 --token YOUR_TOKEN
# Install and start as a service
sudo ./svc.sh install
sudo ./svc.sh start
```
2. **Verify runner environment has**:
- CUDA 11.4+ toolkit installed
- GCC 10 at `/usr/local/bin/gcc` and `/usr/local/bin/g++`
- CMake 3.24+
- Go 1.24+ (or let the workflow install it)
- NVIDIA driver with Tesla K80 support
- Network access to download Go dependencies and models
- **Claude CLI** installed and configured (`claude -p` must be available)
- Install: Follow instructions at https://docs.claude.com/en/docs/claude-code/installation
- The runner needs API access to use Claude for log analysis
3. **Optional: Add runner labels**:
- You can add custom labels like `tesla-k80`, `cuda`, `gpu` during runner configuration
- Then target specific runners by uncommenting the labeled `runs-on` line in the workflow
#### Environment Variables (Optional)
You can set repository secrets or environment variables for:
- `OLLAMA_DEBUG=1` - Enable debug logging
- `OLLAMA_MODELS` - Custom model storage path
- Any other ollama configuration
### Workflow Triggers
The workflow runs on:
- **Push** to `main` or `develop` branches
- **Pull requests** to `main` branch
- **Manual dispatch** via GitHub Actions UI
### Workflow Steps
1. **Environment Setup**: Checkout code, install Go, display system info
2. **Build**: Clean previous builds, configure CMake with GCC 10, build C++/CUDA components and Go binary
3. **Unit Tests**: Run Go unit tests with race detector
4. **Integration Tests**: Start ollama server, wait for ready, run integration tests
5. **Model Tests**: Pull gemma2:2b, run inference, verify GPU acceleration
6. **Log Analysis**: Use Claude headless mode to validate model loaded properly with Tesla K80
7. **Cleanup**: Stop server, upload logs/artifacts
### Artifacts
- **ollama-logs-and-analysis** (always): Server logs, Claude analysis prompt, and analysis result
- **ollama-binary-{sha}** (on success): Compiled ollama binary for the commit
### Log Analysis with Claude
The workflow uses Claude in headless mode (`claude -p`) to intelligently analyze ollama server logs and verify proper Tesla K80 GPU initialization. This provides automated validation that:
1. **Model Loading**: Gemma2:2b loaded without errors
2. **GPU Acceleration**: CUDA properly detected and initialized for Compute 3.7
3. **No CPU Fallback**: Model is running on GPU, not falling back to CPU
4. **No Compatibility Issues**: No CUDA version warnings or errors
5. **Memory Allocation**: Successful GPU memory allocation
6. **Inference Success**: Model inference completed without errors
**Analysis Results**:
- `PASS`: All checks passed, model working correctly with GPU
- `WARN: <reason>`: Model works but has warnings worth reviewing
- `FAIL: <reason>`: Critical issues detected, workflow fails
This approach is superior to simple grep/pattern matching because Claude can:
- Understand context and correlate multiple log entries
- Distinguish between critical errors and benign warnings
- Identify subtle issues like silent CPU fallback
- Provide human-readable explanations of problems
**Example**: If logs show "CUDA initialization successful" but later "using CPU backend", Claude will catch this inconsistency and fail the test, while simple pattern matching might miss it.
### Customization
#### Testing different models
Uncomment and expand the "Test model operations" step:
```yaml
- name: Test model operations
run: |
./ollama pull llama3.2:1b
./ollama run llama3.2:1b "test prompt" --verbose
nvidia-smi # Verify GPU was used
```
#### Running on specific branches
Modify the `on` section:
```yaml
on:
push:
branches: [ main, develop, feature/* ]
```
#### Scheduled runs
Add cron schedule for nightly builds:
```yaml
on:
schedule:
- cron: '0 2 * * *' # 2 AM daily
```
### Troubleshooting
**Runner offline**: Check runner service status
```bash
sudo systemctl status actions.runner.*
```
**Build failures**: Check uploaded logs in Actions > workflow run > Artifacts
**GPU not detected**: Verify `nvidia-smi` works on the runner machine
**Permissions**: Ensure runner user has access to CUDA libraries and can bind to port 11434
### Security Considerations
- Self-hosted runners should be on a secure, isolated machine
- Consider using runner groups to restrict which repositories can use the runner
- Do not use self-hosted runners for public repositories (untrusted PRs)
- Keep the runner software updated

152
.github/workflows/tesla-k80-ci.yml vendored
View File

@@ -1,11 +1,7 @@
name: Tesla K80 Build and Test
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
workflow_dispatch: # Allow manual trigger
workflow_dispatch: # Manual trigger only
jobs:
build-and-test:
@@ -29,157 +25,15 @@ jobs:
- name: Configure CMake
run: |
cmake -B build
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -B build
env:
CMAKE_BUILD_TYPE: Release
- name: Build C++/CUDA components
run: |
cmake --build build --config Release
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake --build build -j$(nproc)
timeout-minutes: 30
- name: Build Go binary
run: |
go build -v -o ollama .
- name: Verify binary
run: |
ls -lh ollama
file ollama
./ollama --version
- name: Run Go unit tests
run: |
go test -v -race -timeout 10m ./...
continue-on-error: false
- name: Start ollama server (background)
run: |
./ollama serve > ollama.log 2>&1 &
echo $! > ollama.pid
echo "Ollama server started with PID $(cat ollama.pid)"
- name: Wait for server to be ready
run: |
for i in {1..30}; do
if curl -s http://localhost:11434/api/tags > /dev/null 2>&1; then
echo "Server is ready!"
exit 0
fi
echo "Waiting for server... attempt $i/30"
sleep 2
done
echo "Server failed to start"
cat ollama.log
exit 1
- name: Run integration tests
run: |
go test -v -timeout 20m ./integration/...
continue-on-error: false
- name: Clear server logs for model test
run: |
# Truncate log file to start fresh for model loading test
> ollama.log
- name: Pull gemma3:4b model
run: |
echo "Pulling gemma3:4b model..."
./ollama pull gemma2:2b
echo "Model pull completed"
timeout-minutes: 15
- name: Run inference with gemma3:4b
run: |
echo "Running inference test..."
./ollama run gemma2:2b "Hello, this is a test. Please respond with a short greeting." --verbose
echo "Inference completed"
timeout-minutes: 5
- name: Wait for logs to flush
run: sleep 3
- name: Analyze server logs with Claude
run: |
echo "Analyzing ollama server logs for proper model loading..."
# Create analysis prompt
cat > log_analysis_prompt.txt << 'EOF'
Analyze the following Ollama server logs from a Tesla K80 (CUDA Compute Capability 3.7) system.
Verify that:
1. The model loaded successfully without errors
2. CUDA/GPU acceleration was properly detected and initialized
3. The model is using the Tesla K80 GPU (not CPU fallback)
4. There are no CUDA compatibility warnings or errors
5. Memory allocation was successful
6. Inference completed without errors
Respond with:
- "PASS" if all checks pass and model loaded properly with GPU acceleration
- "FAIL: <reason>" if there are critical issues
- "WARN: <reason>" if there are warnings but model works
Be specific about what succeeded or failed. Look for CUDA errors, memory issues, or CPU fallback warnings.
Server logs:
---
EOF
cat ollama.log >> log_analysis_prompt.txt
# Run Claude in headless mode to analyze
claude -p log_analysis_prompt.txt > log_analysis_result.txt
echo "=== Claude Analysis Result ==="
cat log_analysis_result.txt
# Check if analysis passed
if grep -q "^PASS" log_analysis_result.txt; then
echo "✓ Log analysis PASSED - Model loaded correctly on Tesla K80"
exit 0
elif grep -q "^WARN" log_analysis_result.txt; then
echo "⚠ Log analysis has WARNINGS - Review needed"
cat log_analysis_result.txt
exit 0 # Don't fail on warnings, but they're visible
else
echo "✗ Log analysis FAILED - Model loading issues detected"
cat log_analysis_result.txt
exit 1
fi
- name: Check GPU memory usage
if: always()
run: |
echo "=== GPU Memory Status ==="
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
- name: Stop ollama server
if: always()
run: |
if [ -f ollama.pid ]; then
kill $(cat ollama.pid) || true
rm ollama.pid
fi
pkill -f "ollama serve" || true
- name: Upload logs and analysis
if: always()
uses: actions/upload-artifact@v4
with:
name: ollama-logs-and-analysis
path: |
ollama.log
log_analysis_prompt.txt
log_analysis_result.txt
build/**/*.log
retention-days: 7
- name: Upload binary artifact
if: success()
uses: actions/upload-artifact@v4
with:
name: ollama-binary-${{ github.sha }}
path: ollama
retention-days: 14

151
.github/workflows/tesla-k80-tests.yml vendored Normal file
View File

@@ -0,0 +1,151 @@
name: Tesla K80 Tests
on:
workflow_dispatch: # Manual trigger only
jobs:
test:
runs-on: self-hosted
timeout-minutes: 60
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Verify ollama binary exists
run: |
if [ ! -f ./ollama ]; then
echo "Error: ollama binary not found. Please run the build workflow first."
exit 1
fi
ls -lh ollama
file ollama
./ollama --version
- name: Run Go unit tests
run: |
go test -v -race -timeout 10m ./...
continue-on-error: false
- name: Start ollama server (background)
run: |
./ollama serve > ollama.log 2>&1 &
echo $! > ollama.pid
echo "Ollama server started with PID $(cat ollama.pid)"
- name: Wait for server to be ready
run: |
for i in {1..30}; do
if curl -s http://localhost:11434/api/tags > /dev/null 2>&1; then
echo "Server is ready!"
exit 0
fi
echo "Waiting for server... attempt $i/30"
sleep 2
done
echo "Server failed to start"
cat ollama.log
exit 1
- name: Run integration tests
run: |
go test -v -timeout 20m ./integration/...
continue-on-error: false
- name: Clear server logs for model test
run: |
# Truncate log file to start fresh for model loading test
> ollama.log
- name: Pull gemma2:2b model
run: |
echo "Pulling gemma2:2b model..."
./ollama pull gemma2:2b
echo "Model pull completed"
timeout-minutes: 15
- name: Run inference with gemma2:2b
run: |
echo "Running inference test..."
./ollama run gemma2:2b "Hello, this is a test. Please respond with a short greeting." --verbose
echo "Inference completed"
timeout-minutes: 5
- name: Wait for logs to flush
run: sleep 3
- name: Analyze server logs with Claude
run: |
echo "Analyzing ollama server logs for proper model loading..."
# Create analysis prompt
cat > log_analysis_prompt.txt << 'EOF'
Analyze the following Ollama server logs from a Tesla K80 (CUDA Compute Capability 3.7) system.
Verify that:
1. The model loaded successfully without errors
2. CUDA/GPU acceleration was properly detected and initialized
3. The model is using the Tesla K80 GPU (not CPU fallback)
4. There are no CUDA compatibility warnings or errors
5. Memory allocation was successful
6. Inference completed without errors
Respond with:
- "PASS" if all checks pass and model loaded properly with GPU acceleration
- "FAIL: <reason>" if there are critical issues
- "WARN: <reason>" if there are warnings but model works
Be specific about what succeeded or failed. Look for CUDA errors, memory issues, or CPU fallback warnings.
Server logs:
---
EOF
cat ollama.log >> log_analysis_prompt.txt
# Run Claude in headless mode to analyze
claude -p log_analysis_prompt.txt > log_analysis_result.txt
echo "=== Claude Analysis Result ==="
cat log_analysis_result.txt
# Check if analysis passed
if grep -q "^PASS" log_analysis_result.txt; then
echo "✓ Log analysis PASSED - Model loaded correctly on Tesla K80"
exit 0
elif grep -q "^WARN" log_analysis_result.txt; then
echo "⚠ Log analysis has WARNINGS - Review needed"
cat log_analysis_result.txt
exit 0 # Don't fail on warnings, but they're visible
else
echo "✗ Log analysis FAILED - Model loading issues detected"
cat log_analysis_result.txt
exit 1
fi
- name: Check GPU memory usage
if: always()
run: |
echo "=== GPU Memory Status ==="
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
- name: Stop ollama server
if: always()
run: |
if [ -f ollama.pid ]; then
kill $(cat ollama.pid) || true
rm ollama.pid
fi
pkill -f "ollama serve" || true
- name: Upload logs and analysis
if: always()
uses: actions/upload-artifact@v4
with:
name: ollama-test-logs-and-analysis
path: |
ollama.log
log_analysis_prompt.txt
log_analysis_result.txt
retention-days: 7